[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:31 dist-test-slave-2hqq krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:31 dist-test-slave-2hqq krb5kdc[399](info): set up 2 sockets
Sep 25 12:11:31 dist-test-slave-2hqq krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 12:11:32.820912 392 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:32 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:32.939668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:33.022189 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:33.022435 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:33.022473 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:33.023627 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:33.023691 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:33.023716 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:33.023746 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:33.023773 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:33.025259 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:33.025563 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:33.025815 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:11:33.031379 412 server_base.cc:1034] running on GCE node
W20260925 12:11:33.031330 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.031343 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.031343 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:33.031836 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:33.032133 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:33.033300 412 hybrid_clock.cc:648] HybridClock initialized: now 1790338293033271 us; error 46 us; skew 500 ppm
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:33.035837 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:33.036285 412 webserver.cc:460] Webserver started at http://127.0.98.62:42409/ using document root <none> and password file <none>
I20260925 12:11:33.036566 412 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:33.036633 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:33.036778 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:33.037868 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "a2b149ad0a9a4c57870e6733849bc2d4"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "31ae248a71cf35133808ef7d9c6fa70f"
server_key_iv: "1cbe528646da3706668343f5f78a5371"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.038256 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2b149ad0a9a4c57870e6733849bc2d4"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "31ae248a71cf35133808ef7d9c6fa70f"
server_key_iv: "1cbe528646da3706668343f5f78a5371"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.039716 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:33.040539 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.040714 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:33.040810 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "a2b149ad0a9a4c57870e6733849bc2d4"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "31ae248a71cf35133808ef7d9c6fa70f"
server_key_iv: "1cbe528646da3706668343f5f78a5371"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.040894 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:33.051092 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:33.053491 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:33.053660 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:33.057590 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45515
I20260925 12:11:33.057631 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45515 every 8 connection(s)
I20260925 12:11:33.057986 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:33.058879 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:33.061266 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4: Bootstrap starting.
I20260925 12:11:33.061818 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:33.062107 480 log.cc:824] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:33.062778 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4: No bootstrap required, opened a new log
I20260925 12:11:33.064323 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } }
I20260925 12:11:33.064525 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:33.064567 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b149ad0a9a4c57870e6733849bc2d4, State: Initialized, Role: FOLLOWER
I20260925 12:11:33.064667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 412
I20260925 12:11:33.064718 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } }
I20260925 12:11:33.064774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:33.064810 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:33.064857 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:33.064913 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:33.065593 392 external_mini_cluster.cc:1428] Setting key 1b840ea05be51f391222c557b6458d25
I20260925 12:11:33.065682 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } }
I20260925 12:11:33.065786 480 leader_election.cc:302] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2b149ad0a9a4c57870e6733849bc2d4; no voters:
I20260925 12:11:33.065951 480 leader_election.cc:288] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:33.065981 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:33.066160 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [term 1 LEADER]: Becoming Leader. State: Replica: a2b149ad0a9a4c57870e6733849bc2d4, State: Running, Role: LEADER
I20260925 12:11:33.066215 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:33.066254 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } }
I20260925 12:11:33.066905 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } } }
I20260925 12:11:33.067001 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:33.067134 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2b149ad0a9a4c57870e6733849bc2d4. Latest consensus state: current_term: 1 leader_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b149ad0a9a4c57870e6733849bc2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45515 } } }
I20260925 12:11:33.067188 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:33.067433 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:33.068209 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:33.069859 494 catalog_manager.cc:1347] Generated new cluster ID: 7208cb134b8d4711a23ad42f2576ab28
I20260925 12:11:33.069916 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:33.080426 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:33.081243 494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:33.083745 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.066733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.0.98.62:45515)
0925 12:11:33.067127 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.067135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.067171 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:11:33.069536 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.069547 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.069591 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.069727 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.071644 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.073134 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.074548 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.075202 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.075325 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.079577 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.079581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.079582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.079593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.081722 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.082268 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.082271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.082271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.082382 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.082667 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.082667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.082668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.082925 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.083061 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.083375 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.083521 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":74,"threads_started":1}
I20260925 12:11:33.085140 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2b149ad0a9a4c57870e6733849bc2d4: Generated new TSK 0
I20260925 12:11:33.085302 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:33.115442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:33.212546 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:33.212769 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:33.212793 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:33.213961 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:33.214004 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:33.214036 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:33.216053 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:33.216321 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:33.216601 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:11:33.222041 506 server_base.cc:1034] running on GCE node
W20260925 12:11:33.222028 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.222020 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.222013 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:33.222429 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:33.222626 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:33.223800 506 hybrid_clock.cc:648] HybridClock initialized: now 1790338293223789 us; error 28 us; skew 500 ppm
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:33.226331 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:33.226753 506 webserver.cc:460] Webserver started at http://127.0.98.1:39587/ using document root <none> and password file <none>
I20260925 12:11:33.226970 506 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:33.227022 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:33.227236 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:33.228364 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "48618ccb9ab94cb28ca2c6a7b5bf7bd0"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "d456e381547aca0bec5e56f5109fcc3d"
server_key_iv: "25dfb2ec10c859d2dc80662918e2b1fd"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.228766 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48618ccb9ab94cb28ca2c6a7b5bf7bd0"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "d456e381547aca0bec5e56f5109fcc3d"
server_key_iv: "25dfb2ec10c859d2dc80662918e2b1fd"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.230072 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:33.230724 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.230868 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:33.230967 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "48618ccb9ab94cb28ca2c6a7b5bf7bd0"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "d456e381547aca0bec5e56f5109fcc3d"
server_key_iv: "25dfb2ec10c859d2dc80662918e2b1fd"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.231087 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:33.241142 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:33.243110 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:33.243292 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:33.243567 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:33.243871 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:33.243923 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.243964 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:33.243992 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.250105 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43799
I20260925 12:11:33.250161 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43799 every 8 connection(s)
I20260925 12:11:33.250495 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:33.251062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 506
I20260925 12:11:33.251153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:33.251343 392 external_mini_cluster.cc:1428] Setting key fe7cc9ab7e50e021c6747cdf3ab5e617
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:11:33.260635 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.251245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43989 (local address 127.0.98.62:45515)
0925 12:11:33.251296 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.251298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.251567 (+ 269us) server_negotiation.cc:407] Connection header received
0925 12:11:33.252150 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.252151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.252165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.252226 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.253342 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.253832 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.254907 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.255248 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.255330 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.257723 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.257724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.257724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.257733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.259118 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.259584 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.259585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.259585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.259647 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.259888 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.259889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.259889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.260003 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.260090 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.260443 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.260549 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:11:33.260768 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.251235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45515 (local address 127.0.98.1:43989)
0925 12:11:33.251418 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:33.251468 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:33.252016 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:33.252299 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:33.252302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:33.252933 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:33.253238 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.253245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.253930 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.253931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.254818 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.254820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.255332 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.255332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.255435 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.255975 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:33.255986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:33.257629 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:33.259220 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.259227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.259229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.259499 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.259710 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.259711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.259712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.259824 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.260074 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:33.260080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:33.260224 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 12:11:33.260331 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 12:11:33.260522 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260925 12:11:33.261272 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45515
I20260925 12:11:33.261379 635 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:33.261557 635 heartbeater.cc:511] Master 127.0.98.62:45515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 12:11:33.262086 444 ts_manager.cc:194] Registered new tserver with Master: 48618ccb9ab94cb28ca2c6a7b5bf7bd0 (127.0.98.1:43799)
I20260925 12:11:33.262722 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:33.276829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:33.362185 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:33.362428 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:33.362478 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:33.363649 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:33.363718 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:33.363765 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:33.365273 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:33.365532 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:33.365757 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:33.370918 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.370884 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.370880 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:33.371017 640 server_base.cc:1034] running on GCE node
I20260925 12:11:33.371345 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:33.371578 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:33.372736 640 hybrid_clock.cc:648] HybridClock initialized: now 1790338293372723 us; error 31 us; skew 500 ppm
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:33.375149 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:33.375581 640 webserver.cc:460] Webserver started at http://127.0.98.2:42681/ using document root <none> and password file <none>
I20260925 12:11:33.375758 640 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:33.375797 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:33.375885 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:33.376968 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "f19a01911c244831aca3e17350b62453"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "55a4ff0dd8c96d7627e38d250a10d6e3"
server_key_iv: "022f24ed4375ec920e6845fc1a77734a"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.377274 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f19a01911c244831aca3e17350b62453"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "55a4ff0dd8c96d7627e38d250a10d6e3"
server_key_iv: "022f24ed4375ec920e6845fc1a77734a"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.378525 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:33.379233 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.379434 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:33.379530 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "f19a01911c244831aca3e17350b62453"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "55a4ff0dd8c96d7627e38d250a10d6e3"
server_key_iv: "022f24ed4375ec920e6845fc1a77734a"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.379587 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:33.395641 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:33.397707 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:33.397833 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:33.398096 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:33.398367 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:33.398399 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.398423 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:33.398437 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.405671 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43977
I20260925 12:11:33.405720 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43977 every 8 connection(s)
I20260925 12:11:33.406064 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:33.411936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 640
I20260925 12:11:33.412042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:33.412252 392 external_mini_cluster.cc:1428] Setting key 7f8ed527f2e3475c0dc9a70f203afcc9
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:33.416581 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.406780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35925 (local address 127.0.98.62:45515)
0925 12:11:33.406866 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.406868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.407127 (+ 259us) server_negotiation.cc:407] Connection header received
0925 12:11:33.407677 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.407679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.407695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.407757 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.408782 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.409344 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.410433 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.410671 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.410763 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.413438 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.413439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.413439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.413445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.415164 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.415583 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.415585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.415585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.415648 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.415895 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.415896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.415896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.416015 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.416085 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.416362 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.416448 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:11:33.416771 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.406741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45515 (local address 127.0.98.2:35925)
0925 12:11:33.406921 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:33.407031 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:33.407563 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:33.407837 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:33.407841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:33.408448 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:33.408678 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.408683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.409456 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.409457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.410332 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.410334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.410774 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.410775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.410904 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.411478 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:33.411487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:33.413352 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:33.415249 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.415254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.415256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.415472 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.415716 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.415717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.415717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.415823 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.416091 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:33.416097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:33.416229 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 12:11:33.416328 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 12:11:33.416482 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260925 12:11:33.417197 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45515
I20260925 12:11:33.417284 769 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:33.417471 769 heartbeater.cc:511] Master 127.0.98.62:45515 requested a full tablet report, sending...
I20260925 12:11:33.417840 444 ts_manager.cc:194] Registered new tserver with Master: f19a01911c244831aca3e17350b62453 (127.0.98.2:43977)
I20260925 12:11:33.418490 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35925
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:33.438946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:33.520619 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:33.520818 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:33.520841 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:33.521919 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:33.521955 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:33.521981 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:33.523381 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:33.523578 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:33.523746 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:33.529023 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:33.529126 774 server_base.cc:1034] running on GCE node
W20260925 12:11:33.528976 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:33.528991 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:33.529421 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:33.529662 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:33.530835 774 hybrid_clock.cc:648] HybridClock initialized: now 1790338293530825 us; error 29 us; skew 500 ppm
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:33.533548 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:33.534051 774 webserver.cc:460] Webserver started at http://127.0.98.3:46095/ using document root <none> and password file <none>
I20260925 12:11:33.534296 774 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:33.534340 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:33.534487 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:33.535578 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "bba5e45aadbd4c0c99a45313b5b210ae"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "8c439bb3e4bdca7ca004cbf01a12423f"
server_key_iv: "9522a92f2d929971d37b9ce3e84f5906"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.535934 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bba5e45aadbd4c0c99a45313b5b210ae"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "8c439bb3e4bdca7ca004cbf01a12423f"
server_key_iv: "9522a92f2d929971d37b9ce3e84f5906"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.537361 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:33.538280 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.538472 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:33.538551 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "bba5e45aadbd4c0c99a45313b5b210ae"
format_stamp: "Formatted at 2026-09-25 12:11:33 on dist-test-slave-2hqq"
server_key: "8c439bb3e4bdca7ca004cbf01a12423f"
server_key_iv: "9522a92f2d929971d37b9ce3e84f5906"
server_key_version: "encryptionkey@0"
I20260925 12:11:33.538606 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:33.575830 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:33.578179 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:33.578297 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:33.578495 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:33.578814 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:33.578850 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.578893 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:33.578919 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:33.585966 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34129
I20260925 12:11:33.586035 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34129 every 8 connection(s)
I20260925 12:11:33.586346 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:33.594098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 774
I20260925 12:11:33.594184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:33.594357 392 external_mini_cluster.cc:1428] Setting key a669b199ce97e0568a2ee1da30386815
I20260925 12:11:33.596194 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.587115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60131 (local address 127.0.98.62:45515)
0925 12:11:33.587188 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.587190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.587341 (+ 151us) server_negotiation.cc:407] Connection header received
0925 12:11:33.587790 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.587792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.587802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.587873 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.589031 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.589582 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.590668 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.591091 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.591176 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.593378 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.593379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.593380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.593386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.594914 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.595405 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.595406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.595406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.595456 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.595690 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.595691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.595691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.595798 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.595849 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.596070 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.596130 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:11:33.596346 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.587044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45515 (local address 127.0.98.3:60131)
0925 12:11:33.587196 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:33.587238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:33.587709 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:33.587932 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:33.587936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:33.588571 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:33.588894 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.588901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.589666 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.589666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.590584 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:33.590586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.591186 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.591187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:33.591290 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.591841 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:33.591849 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:33.593296 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:33.594986 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.594991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.594993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.595339 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.595529 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:33.595530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:33.595534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:33.595634 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:33.595840 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:33.595843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:33.595951 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 12:11:33.596026 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:11:33.596151 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260925 12:11:33.596812 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45515
I20260925 12:11:33.596911 903 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:33.597081 903 heartbeater.cc:511] Master 127.0.98.62:45515 requested a full tablet report, sending...
I20260925 12:11:33.597420 444 ts_manager.cc:194] Registered new tserver with Master: bba5e45aadbd4c0c99a45313b5b210ae (127.0.98.3:34129)
I20260925 12:11:33.597781 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60131
I20260925 12:11:33.606475 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:33.630648 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.621878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.0.98.1:43799)
0925 12:11:33.622061 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.622067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.622141 (+ 74us) server_negotiation.cc:407] Connection header received
0925 12:11:33.622239 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.622243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.622262 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.622356 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.622959 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.623753 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.624618 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.624967 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.625065 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.627499 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.627507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.627508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.627516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.629397 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.629784 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.629788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.629789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.629851 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.630061 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.630062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.630062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.630207 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.630373 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.630428 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.630515 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
W20260925 12:11:33.631076 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53460
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:33.653126 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:33.646127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.1:43799)
0925 12:11:33.646192 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:33.646194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:33.646248 (+ 54us) server_negotiation.cc:407] Connection header received
0925 12:11:33.646305 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:33.646306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:33.646318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:33.646368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:33.646902 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.647409 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.648166 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:33.648357 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:33.648407 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:33.650571 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:33.650573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:33.650573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:33.650580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:33.652032 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.652364 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.652364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.652365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.652413 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:33.652725 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:33.652726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:33.652726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:33.652816 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:33.652907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:33.652975 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 12:11:33.653036 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:11:33.654165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 506
I20260925 12:11:33.658806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 640
I20260925 12:11:33.664584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 774
I20260925 12:11:33.669713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 412
2026-09-25T12:11:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2678 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[917](info): set up 2 sockets
Sep 25 12:11:33 dist-test-slave-2hqq krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 12:11:34.708068 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:34 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:34.823319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:34.904002 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:34.904243 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:34.904282 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:34.905450 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:34.905521 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:34.905545 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:34.905575 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:34.905604 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:34.907037 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:34.907303 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:34.907527 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:34.912364 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:34.912427 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:34.912431 930 server_base.cc:1034] running on GCE node
W20260925 12:11:34.912359 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:34.912771 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:34.913054 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:34.914201 930 hybrid_clock.cc:648] HybridClock initialized: now 1790338294914187 us; error 31 us; skew 500 ppm
Sep 25 12:11:34 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:34.916739 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:34.917243 930 webserver.cc:460] Webserver started at http://127.0.98.62:37181/ using document root <none> and password file <none>
I20260925 12:11:34.917486 930 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:34.917532 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:34.917675 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:34.918711 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "887952a9357e4127b87037993ad86ab4"
format_stamp: "Formatted at 2026-09-25 12:11:34 on dist-test-slave-2hqq"
server_key: "a694a4b13c9b5e94d8eb26639fa4a45d"
server_key_iv: "f31f5229a1da4f20928b68271e87fe0e"
server_key_version: "encryptionkey@0"
I20260925 12:11:34.919059 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "887952a9357e4127b87037993ad86ab4"
format_stamp: "Formatted at 2026-09-25 12:11:34 on dist-test-slave-2hqq"
server_key: "a694a4b13c9b5e94d8eb26639fa4a45d"
server_key_iv: "f31f5229a1da4f20928b68271e87fe0e"
server_key_version: "encryptionkey@0"
I20260925 12:11:34.920332 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:34.921188 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:34.921350 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:34.921445 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "887952a9357e4127b87037993ad86ab4"
format_stamp: "Formatted at 2026-09-25 12:11:34 on dist-test-slave-2hqq"
server_key: "a694a4b13c9b5e94d8eb26639fa4a45d"
server_key_iv: "f31f5229a1da4f20928b68271e87fe0e"
server_key_version: "encryptionkey@0"
I20260925 12:11:34.921522 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:34.935959 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:34.938396 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:34.938565 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:34.942721 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34035
I20260925 12:11:34.942759 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34035 every 8 connection(s)
I20260925 12:11:34.943132 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:34.944032 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:34.946342 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: Bootstrap starting.
I20260925 12:11:34.947089 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:34.947357 998 log.cc:824] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:34.948020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 930
I20260925 12:11:34.948110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:34.948120 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: No bootstrap required, opened a new log
I20260925 12:11:34.948277 392 external_mini_cluster.cc:1428] Setting key 8cbe8e9b16b174bef2c10c49b58e8e77
I20260925 12:11:34.950318 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260925 12:11:34.950474 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:34.950521 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887952a9357e4127b87037993ad86ab4, State: Initialized, Role: FOLLOWER
I20260925 12:11:34.950685 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260925 12:11:34.950774 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:34.950814 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:34.950855 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:34.951399 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260925 12:11:34.951514 998 leader_election.cc:302] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 887952a9357e4127b87037993ad86ab4; no voters:
I20260925 12:11:34.951689 998 leader_election.cc:288] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:34.951733 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:34.951857 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 887952a9357e4127b87037993ad86ab4, State: Running, Role: LEADER
I20260925 12:11:34.951936 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260925 12:11:34.951975 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:34.952251 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 887952a9357e4127b87037993ad86ab4. Latest consensus state: current_term: 1 leader_uuid: "887952a9357e4127b87037993ad86ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } } }
I20260925 12:11:34.952319 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:34.952275 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "887952a9357e4127b87037993ad86ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887952a9357e4127b87037993ad86ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } } }
I20260925 12:11:34.952533 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4 [sys.catalog]: This master's current role is: LEADER
W20260925 12:11:34.953742 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:11:34.953833 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:11:34.954089 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:34.954260 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:11:34 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:34.956804 1008 catalog_manager.cc:1347] Generated new cluster ID: 9805530315c54fad90bb8ea1d5a4dad5
I20260925 12:11:34.956861 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:34.960232 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:34.948927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.0.98.62:34035)
0925 12:11:34.949125 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:34.949136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:34.949169 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:11:34.949899 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:34.949909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:34.949976 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:34.950141 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:34.950907 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:34.952396 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:34.953561 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:34.954196 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:34.954304 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:34.956956 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:34.956960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:34.956960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:34.956972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:34.958657 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:34.959029 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:34.959032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:34.959033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:34.959119 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:34.959325 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:34.959325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:34.959326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:34.959565 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:34.959740 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:34.959823 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:11:34.960077 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260925 12:11:34.994817 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:34.995394 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:34.999431 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 887952a9357e4127b87037993ad86ab4: Generated new TSK 0
I20260925 12:11:34.999625 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:35.029537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:35.114665 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:35.114872 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:35.114897 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:35.116014 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:35.116052 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:35.116079 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:35.117498 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:35.117703 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:35.117892 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:35.122977 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:35.122979 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:35.123031 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:35.123342 1024 server_base.cc:1034] running on GCE node
I20260925 12:11:35.123543 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:35.123754 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:35.124933 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790338295124923 us; error 30 us; skew 500 ppm
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:35.127651 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:35.128078 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35591/ using document root <none> and password file <none>
I20260925 12:11:35.128281 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:35.128345 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:35.128443 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:35.129539 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "0814f764a0974916ba0318ca74ee68a9"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "bdf4d0aa269ab8cbdfb47b4aba1e25cb"
server_key_iv: "e1b56951752f0dd3f7c95943d549c36d"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.129842 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0814f764a0974916ba0318ca74ee68a9"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "bdf4d0aa269ab8cbdfb47b4aba1e25cb"
server_key_iv: "e1b56951752f0dd3f7c95943d549c36d"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.131124 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:35.131963 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.132156 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:35.132243 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "0814f764a0974916ba0318ca74ee68a9"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "bdf4d0aa269ab8cbdfb47b4aba1e25cb"
server_key_iv: "e1b56951752f0dd3f7c95943d549c36d"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.132301 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:35.154421 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:35.156759 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:35.156883 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:35.157083 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:35.157421 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:35.157459 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.157482 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:35.157529 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.163116 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41849
I20260925 12:11:35.163276 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41849 every 8 connection(s)
I20260925 12:11:35.163568 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:35.164979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1024
I20260925 12:11:35.165076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:35.165242 392 external_mini_cluster.cc:1428] Setting key 97defa800cb092e1f59e516090340fe1
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:11:35.174937 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.164412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34035 (local address 127.0.98.1:35429)
0925 12:11:35.164612 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.164662 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.165279 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.165537 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.165541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.166178 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:35.166373 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.166377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.167417 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.167419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.168746 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.168749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.169289 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.169290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.169398 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.169973 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:35.169981 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:35.171640 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:35.173603 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.173609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.173611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.173828 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.174037 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.174037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.174038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.174136 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.174432 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:35.174437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:35.174559 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.174647 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.174771 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260925 12:11:35.174835 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.164521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35429 (local address 127.0.98.62:34035)
0925 12:11:35.164578 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.164580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.164739 (+ 159us) server_negotiation.cc:407] Connection header received
0925 12:11:35.165389 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.165391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.165403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.165455 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.166475 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.167293 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.168831 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.169193 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.169275 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.171737 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.171738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.171738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.171747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.173517 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.173904 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.173905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.173905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.173971 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.174201 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.174202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.174203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.174368 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.174443 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.174691 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.174765 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:11:35.175596 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34035
I20260925 12:11:35.175694 1153 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:35.175849 1153 heartbeater.cc:511] Master 127.0.98.62:34035 requested a full tablet report, sending...
I20260925 12:11:35.176385 962 ts_manager.cc:194] Registered new tserver with Master: 0814f764a0974916ba0318ca74ee68a9 (127.0.98.1:41849)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 12:11:35.177287 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:35.191745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:35.273911 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:35.274124 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:35.274147 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:35.275259 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:35.275297 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:35.275322 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:35.276746 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:35.276935 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:35.277115 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:35.282073 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:35.282212 1158 server_base.cc:1034] running on GCE node
W20260925 12:11:35.282075 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:35.282133 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:35.282557 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:35.282815 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:35.283983 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790338295283979 us; error 36 us; skew 500 ppm
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:35.286643 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:35.287127 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43025/ using document root <none> and password file <none>
I20260925 12:11:35.287420 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:35.287612 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:35.287770 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:35.288987 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "254843bb011c4be4af143f44c282fd45"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "3d13004f1d3a378e4f87d77587ca7bcb"
server_key_iv: "16b31845d91835ea2dbeab2d9a3bc4b1"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.289374 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "254843bb011c4be4af143f44c282fd45"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "3d13004f1d3a378e4f87d77587ca7bcb"
server_key_iv: "16b31845d91835ea2dbeab2d9a3bc4b1"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.290673 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:35.291395 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.291579 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:35.291676 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "254843bb011c4be4af143f44c282fd45"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "3d13004f1d3a378e4f87d77587ca7bcb"
server_key_iv: "16b31845d91835ea2dbeab2d9a3bc4b1"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.291747 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:35.310869 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:35.313113 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:35.313278 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:35.313548 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:35.313918 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:35.313975 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.314018 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:35.314051 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.319933 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44609
I20260925 12:11:35.319984 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44609 every 8 connection(s)
I20260925 12:11:35.320353 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:35.326555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1158
I20260925 12:11:35.326645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:35.326807 392 external_mini_cluster.cc:1428] Setting key 17392a6537101da465adfd5fade051e1
I20260925 12:11:35.329957 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.321090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34035 (local address 127.0.98.2:34817)
0925 12:11:35.321246 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.321281 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.321797 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.322005 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.322008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.322625 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:35.322825 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.322830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.323601 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.323602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.324551 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.324553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.324907 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.324909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.325011 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.325504 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:35.325512 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:35.326868 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:35.328662 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.328668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.328670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.328904 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.329107 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.329107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.329108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.329202 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.329469 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:35.329472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:35.329586 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.329669 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.329786 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20260925 12:11:35.329953 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.321127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34817 (local address 127.0.98.62:34035)
0925 12:11:35.321203 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.321205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.321383 (+ 178us) server_negotiation.cc:407] Connection header received
0925 12:11:35.321882 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.321883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.321893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.321944 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.322932 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.323490 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.324640 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.324832 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.324894 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.326961 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.326962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.326963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.326971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.328583 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.328977 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.328978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.328979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.329043 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.329261 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.329262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.329262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.329401 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.329478 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.329713 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.329804 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:11:35.330391 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34035
I20260925 12:11:35.330508 1287 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:35.330687 1287 heartbeater.cc:511] Master 127.0.98.62:34035 requested a full tablet report, sending...
I20260925 12:11:35.331082 962 ts_manager.cc:194] Registered new tserver with Master: 254843bb011c4be4af143f44c282fd45 (127.0.98.2:44609)
I20260925 12:11:35.331593 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34817
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:35.355058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:35.436662 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:35.436915 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:35.436959 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:35.438230 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:35.438305 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:35.438357 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:35.439925 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:35.440196 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:35.440433 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:35.445580 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:35.445573 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:35.445573 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:35.445753 1292 server_base.cc:1034] running on GCE node
I20260925 12:11:35.445961 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:35.446162 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:35.447294 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790338295447279 us; error 29 us; skew 500 ppm
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:35.449820 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:35.450258 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37743/ using document root <none> and password file <none>
I20260925 12:11:35.450485 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:35.450532 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:35.450659 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:35.451815 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "f3b81c145ddab9e56792e9f0921cebe6"
server_key_iv: "a17045af5b61d00b7fa50f72192d5e08"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.452169 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "f3b81c145ddab9e56792e9f0921cebe6"
server_key_iv: "a17045af5b61d00b7fa50f72192d5e08"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.453715 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:35.454550 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.454824 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:35.454936 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e"
format_stamp: "Formatted at 2026-09-25 12:11:35 on dist-test-slave-2hqq"
server_key: "f3b81c145ddab9e56792e9f0921cebe6"
server_key_iv: "a17045af5b61d00b7fa50f72192d5e08"
server_key_version: "encryptionkey@0"
I20260925 12:11:35.455008 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:35.488304 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:35.490727 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:35.490890 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:35.491129 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:35.491436 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:35.491487 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.491528 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:35.491559 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:35.497780 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33399
I20260925 12:11:35.497818 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33399 every 8 connection(s)
I20260925 12:11:35.498142 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:35.500121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1292
I20260925 12:11:35.500216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:35.500381 392 external_mini_cluster.cc:1428] Setting key d992363e77f093cf4db8c3dab836c1cc
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:35.507949 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.498910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52691 (local address 127.0.98.62:34035)
0925 12:11:35.498965 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.498967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.499141 (+ 174us) server_negotiation.cc:407] Connection header received
0925 12:11:35.500206 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.500208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.500222 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.500291 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.501213 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.501738 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.502742 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.502942 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.503023 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.505357 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.505359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.505359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.505367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.506562 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.506993 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.506995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.506995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.507050 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.507312 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.507313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.507313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.507427 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.507501 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.507778 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.507855 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:11:35.508149 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.498886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34035 (local address 127.0.98.3:52691)
0925 12:11:35.499026 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.499061 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.500045 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.500425 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.500430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.500870 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:35.501126 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.501131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.501837 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.501838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.502655 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.502656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.503030 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.503031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.503167 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.503674 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:35.503684 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:35.505257 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:35.506659 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.506666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.506668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.506905 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.507113 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:35.507114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:35.507115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:35.507224 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:35.507491 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:35.507497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:35.507638 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.507751 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.507913 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260925 12:11:35.508635 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34035
I20260925 12:11:35.508733 1421 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:35.508922 1421 heartbeater.cc:511] Master 127.0.98.62:34035 requested a full tablet report, sending...
I20260925 12:11:35.509344 962 ts_manager.cc:194] Registered new tserver with Master: f74304d87fc44caa9bd3aa7b75c8dc9e (127.0.98.3:33399)
I20260925 12:11:35.509706 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52691
I20260925 12:11:35.512099 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:11:35.519510 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.513640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.62:34035)
0925 12:11:35.513725 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.513727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.513802 (+ 75us) server_negotiation.cc:407] Connection header received
0925 12:11:35.513822 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.513822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.513831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.513880 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.514417 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.514939 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.515636 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.515861 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.515929 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.517135 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.517137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.517137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.517144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.518398 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.518738 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.518740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.518740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.518814 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.519037 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.519038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.519038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.519183 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.519260 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.519362 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.519444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 12:11:35.521891 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:35.523464 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:35.530378 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.526298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.98.1:41849)
0925 12:11:35.526476 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.526480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.526510 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:11:35.526575 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.526577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.526592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.526666 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:35.526918 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.527949 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.529358 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.530051 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.530116 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530163 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.530257 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.530296 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260925 12:11:35.530395 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.525965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41849 (local address 127.0.0.1:57758)
0925 12:11:35.526452 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.526462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.526520 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.526741 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.526742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.526744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:35.526783 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.526784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.528035 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.528036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.529274 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.529277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.530114 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.530115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.530176 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530180 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.530239 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.530280 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":19,"thread_start_us":56,"threads_started":1}
I20260925 12:11:35.530579 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.526210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.3:33399)
0925 12:11:35.526384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.526388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.526641 (+ 253us) server_negotiation.cc:407] Connection header received
0925 12:11:35.526712 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.526714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.526728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.526833 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:35.526970 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.527754 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.529379 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.529890 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.529982 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530044 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.530437 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.530487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260925 12:11:35.530592 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.526118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.2:44609)
0925 12:11:35.526284 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.526288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.526299 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:35.526336 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.526338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.526352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.526423 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:35.526666 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.527790 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.529214 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.529948 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.530023 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530080 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.530446 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.530499 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 12:11:35.530396 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.525965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44609 (local address 127.0.0.1:46464)
0925 12:11:35.526131 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.526170 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.526237 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.526512 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.526516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.526518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:35.526582 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.526585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.527918 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.527919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.529135 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.529137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.530117 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.530118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.530192 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530195 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.530245 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.530292 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":85,"thread_start_us":100,"threads_started":1}
I20260925 12:11:35.530722 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.525980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33399 (local address 127.0.0.1:55440)
0925 12:11:35.526581 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.526588 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.526642 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.526833 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.526834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.526836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:35.526883 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.526884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.527857 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.527858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.529275 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.529277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.529981 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.529982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.530057 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.530071 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.530215 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.530280 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":82,"thread_start_us":92,"threads_started":1}
I20260925 12:11:35.531498 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9b7d96990545e0b9e628137e4d0c79 (DEFAULT_TABLE table=test-table [id=92f921b884324a2286061ea912e246b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:35.531524 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9b7d96990545e0b9e628137e4d0c79 (DEFAULT_TABLE table=test-table [id=92f921b884324a2286061ea912e246b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:35.531503 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9b7d96990545e0b9e628137e4d0c79 (DEFAULT_TABLE table=test-table [id=92f921b884324a2286061ea912e246b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:35.531847 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9b7d96990545e0b9e628137e4d0c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:35.531847 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9b7d96990545e0b9e628137e4d0c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:35.531847 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9b7d96990545e0b9e628137e4d0c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:35.534740 1438 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Bootstrap starting.
I20260925 12:11:35.535132 1440 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Bootstrap starting.
I20260925 12:11:35.535611 1438 tablet_bootstrap.cc:654] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:35.535748 1440 tablet_bootstrap.cc:654] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:35.536258 1438 log.cc:824] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:35.536322 1440 log.cc:824] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:35.536638 1439 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Bootstrap starting.
I20260925 12:11:35.537202 1438 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: No bootstrap required, opened a new log
I20260925 12:11:35.537251 1439 tablet_bootstrap.cc:654] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:35.537309 1438 ts_tablet_manager.cc:1397] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 12:11:35.537815 1439 log.cc:824] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:35.538642 1439 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: No bootstrap required, opened a new log
I20260925 12:11:35.538715 1439 ts_tablet_manager.cc:1397] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:35.539384 1438 raft_consensus.cc:348] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.539592 1438 raft_consensus.cc:374] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:35.539629 1438 raft_consensus.cc:729] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74304d87fc44caa9bd3aa7b75c8dc9e, State: Initialized, Role: FOLLOWER
I20260925 12:11:35.539744 1438 consensus_queue.cc:260] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.540024 1438 ts_tablet_manager.cc:1428] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:11:35.540052 1421 heartbeater.cc:503] Master 127.0.98.62:34035 was elected leader, sending a full tablet report...
I20260925 12:11:35.540719 1439 raft_consensus.cc:348] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.540894 1439 raft_consensus.cc:374] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:35.540946 1439 raft_consensus.cc:729] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0814f764a0974916ba0318ca74ee68a9, State: Initialized, Role: FOLLOWER
I20260925 12:11:35.540990 1440 tablet_bootstrap.cc:492] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: No bootstrap required, opened a new log
I20260925 12:11:35.541077 1440 ts_tablet_manager.cc:1397] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 12:11:35.541075 1439 consensus_queue.cc:260] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.541348 1439 ts_tablet_manager.cc:1428] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:11:35.541358 1153 heartbeater.cc:503] Master 127.0.98.62:34035 was elected leader, sending a full tablet report...
I20260925 12:11:35.543038 1440 raft_consensus.cc:348] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.543188 1440 raft_consensus.cc:374] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:35.543211 1440 raft_consensus.cc:729] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254843bb011c4be4af143f44c282fd45, State: Initialized, Role: FOLLOWER
I20260925 12:11:35.543365 1440 consensus_queue.cc:260] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.543668 1287 heartbeater.cc:503] Master 127.0.98.62:34035 was elected leader, sending a full tablet report...
I20260925 12:11:35.543665 1440 ts_tablet_manager.cc:1428] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 12:11:35.570963 1288 tablet.cc:2369] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:11:35.664317 1154 tablet.cc:2369] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:11:35.748759 1422 tablet.cc:2369] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:35.934170 1445 raft_consensus.cc:482] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:35.934321 1445 raft_consensus.cc:504] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.934692 1445 leader_election.cc:288] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 254843bb011c4be4af143f44c282fd45 (127.0.98.2:44609), f74304d87fc44caa9bd3aa7b75c8dc9e (127.0.98.3:33399)
I20260925 12:11:35.938973 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.934879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44609 (local address 127.0.98.1:58133)
0925 12:11:35.935051 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.935064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.935152 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.935321 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.935321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.935348 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:35.935425 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.935427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.936347 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.936348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.937813 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.937816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.938634 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.938636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.938771 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.938774 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.938830 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.938859 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":36,"thread_start_us":77,"threads_started":1}
I20260925 12:11:35.939019 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.934980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58133 (local address 127.0.98.2:44609)
0925 12:11:35.935031 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.935033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.935157 (+ 124us) server_negotiation.cc:407] Connection header received
0925 12:11:35.935223 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.935224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.935236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.935293 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:35.935511 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.936271 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.937943 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.938526 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.938607 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.938618 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.938880 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.938950 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:11:35.939088 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.935029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33019 (local address 127.0.98.3:33399)
0925 12:11:35.935084 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.935086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.935198 (+ 112us) server_negotiation.cc:407] Connection header received
0925 12:11:35.935223 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.935223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.935236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.935293 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:35.935511 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.936320 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.937943 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.938530 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.938607 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.938618 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.938977 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.939019 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:11:35.938972 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.934964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33399 (local address 127.0.98.1:33019)
0925 12:11:35.935083 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:35.935092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:35.935152 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:35.935321 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:35.935321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:35.935342 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:35.935424 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.935428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.936519 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.936521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.937813 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:35.937816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.938636 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.938636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:35.938730 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.938734 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:35.938803 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:11:35.938853 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":66,"thread_start_us":77,"threads_started":1}
I20260925 12:11:35.939604 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9b7d96990545e0b9e628137e4d0c79" candidate_uuid: "0814f764a0974916ba0318ca74ee68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254843bb011c4be4af143f44c282fd45" is_pre_election: true
I20260925 12:11:35.939630 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9b7d96990545e0b9e628137e4d0c79" candidate_uuid: "0814f764a0974916ba0318ca74ee68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" is_pre_election: true
I20260925 12:11:35.939790 1241 raft_consensus.cc:2454] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0814f764a0974916ba0318ca74ee68a9 in term 0.
I20260925 12:11:35.939790 1375 raft_consensus.cc:2454] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0814f764a0974916ba0318ca74ee68a9 in term 0.
I20260925 12:11:35.940042 1040 leader_election.cc:302] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0814f764a0974916ba0318ca74ee68a9, 254843bb011c4be4af143f44c282fd45; no voters:
I20260925 12:11:35.940223 1445 raft_consensus.cc:2781] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:35.940264 1445 raft_consensus.cc:482] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:35.940280 1445 raft_consensus.cc:3037] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:35.941016 1445 raft_consensus.cc:504] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.941171 1445 leader_election.cc:288] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [CANDIDATE]: Term 1 election: Requested vote from peers 254843bb011c4be4af143f44c282fd45 (127.0.98.2:44609), f74304d87fc44caa9bd3aa7b75c8dc9e (127.0.98.3:33399)
I20260925 12:11:35.941327 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9b7d96990545e0b9e628137e4d0c79" candidate_uuid: "0814f764a0974916ba0318ca74ee68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254843bb011c4be4af143f44c282fd45"
I20260925 12:11:35.941372 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9b7d96990545e0b9e628137e4d0c79" candidate_uuid: "0814f764a0974916ba0318ca74ee68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e"
I20260925 12:11:35.941447 1241 raft_consensus.cc:3037] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:35.941445 1375 raft_consensus.cc:3037] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:35.942173 1241 raft_consensus.cc:2454] T 3a9b7d96990545e0b9e628137e4d0c79 P 254843bb011c4be4af143f44c282fd45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0814f764a0974916ba0318ca74ee68a9 in term 1.
I20260925 12:11:35.942376 1040 leader_election.cc:302] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0814f764a0974916ba0318ca74ee68a9, 254843bb011c4be4af143f44c282fd45; no voters:
I20260925 12:11:35.942462 1375 raft_consensus.cc:2454] T 3a9b7d96990545e0b9e628137e4d0c79 P f74304d87fc44caa9bd3aa7b75c8dc9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0814f764a0974916ba0318ca74ee68a9 in term 1.
I20260925 12:11:35.942524 1445 raft_consensus.cc:2781] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:35.942699 1445 raft_consensus.cc:686] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0814f764a0974916ba0318ca74ee68a9, State: Running, Role: LEADER
I20260925 12:11:35.942803 1445 consensus_queue.cc:237] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } }
I20260925 12:11:35.943691 961 catalog_manager.cc:5495] T 3a9b7d96990545e0b9e628137e4d0c79 P 0814f764a0974916ba0318ca74ee68a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0814f764a0974916ba0318ca74ee68a9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0814f764a0974916ba0318ca74ee68a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0814f764a0974916ba0318ca74ee68a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "254843bb011c4be4af143f44c282fd45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74304d87fc44caa9bd3aa7b75c8dc9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33399 } health_report { overall_health: UNKNOWN } } }
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:35.975458 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.966974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.1:41849)
0925 12:11:35.967048 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.967051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.967123 (+ 72us) server_negotiation.cc:407] Connection header received
0925 12:11:35.967204 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.967205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.967217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.967275 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.967829 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.968316 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.969163 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.969398 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.969483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.972439 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.972461 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.972461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.972469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.974100 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.974601 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.974606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.974606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.974669 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.974878 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.974879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.974880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.975031 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.975215 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.975259 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.975337 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:11:35 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:36.000021 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:35.992615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.0.98.1:41849)
0925 12:11:35.992697 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:35.992700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:35.992831 (+ 131us) server_negotiation.cc:407] Connection header received
0925 12:11:35.992874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:35.992875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:35.992885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:35.992939 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:35.993506 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.994002 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.994919 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:35.995113 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:35.995178 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:35.997549 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:35.997550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:35.997551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:35.997558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:35.998799 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.999242 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.999244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.999244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.999316 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:35.999494 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:35.999495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:35.999495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:35.999590 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:35.999664 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:35.999826 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 12:11:35.999892 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 12:11:36 dist-test-slave-2hqq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:11:36 dist-test-slave-2hqq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:36.022603 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:36.015488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.98.1:41849)
0925 12:11:36.015559 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:36.015561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:36.015572 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:36.015603 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:36.015604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:36.015613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:36.015685 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:36.016253 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:36.016880 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:36.017640 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:36.017879 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:36.017941 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:36.020340 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:36.020341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:36.020342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:36.020349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:36.021652 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:36.021981 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:36.021982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:36.021982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:36.022034 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:36.022200 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:36.022200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:36.022201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:36.022290 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:36.022365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:36.022473 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:11:36.022535 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 12:11:36.023100 1067 tablet_service.cc:2908] Rejecting scan request for tablet 3a9b7d96990545e0b9e628137e4d0c79: Uninitialized: clean time has not yet been initialized
I20260925 12:11:36.023874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1024
I20260925 12:11:36.029415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1158
I20260925 12:11:36.034159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1292
I20260925 12:11:36.038443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 930
2026-09-25T12:11:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2370 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:36 dist-test-slave-2hqq krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:36 dist-test-slave-2hqq krb5kdc[1463](info): set up 2 sockets
Sep 25 12:11:36 dist-test-slave-2hqq krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260925 12:11:37.148525 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:37.343436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:37.424185 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:37.424433 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:37.424468 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:37.425621 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:37.425694 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:11:37.425716 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:37.425740 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:37.425768 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:37.425792 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:37.427286 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44637
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:37.427556 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:37.427778 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:37.432480 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.432480 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.432487 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:37.432717 1476 server_base.cc:1034] running on GCE node
I20260925 12:11:37.433179 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:37.433470 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:37.434653 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790338297434645 us; error 41 us; skew 500 ppm
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:37.437256 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:37.437708 1476 webserver.cc:460] Webserver started at http://127.0.98.62:45707/ using document root <none> and password file <none>
I20260925 12:11:37.437992 1476 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:37.438059 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:37.438232 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:37.439280 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "20260af241334fd5bd6972464b8d1e55"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "4df26e148aff6f9985577d726acc0c52"
server_key_iv: "b3f2f91e7c0fdf9b089213177ad85108"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.439639 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "20260af241334fd5bd6972464b8d1e55"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "4df26e148aff6f9985577d726acc0c52"
server_key_iv: "b3f2f91e7c0fdf9b089213177ad85108"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.441092 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:37.441854 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.442019 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:37.442137 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "20260af241334fd5bd6972464b8d1e55"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "4df26e148aff6f9985577d726acc0c52"
server_key_iv: "b3f2f91e7c0fdf9b089213177ad85108"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.442209 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:37.474242 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:37.476568 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:37.476745 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:37.482278 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44637
I20260925 12:11:37.482867 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:37.483754 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:37.484705 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44637 every 8 connection(s)
I20260925 12:11:37.486754 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Bootstrap starting.
I20260925 12:11:37.487440 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:37.487764 1554 log.cc:824] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:37.488606 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: No bootstrap required, opened a new log
I20260925 12:11:37.488755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1476
I20260925 12:11:37.488862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:37.489014 392 external_mini_cluster.cc:1428] Setting key 67d8443ea0d545b3af7d575840e62678
I20260925 12:11:37.491000 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } }
I20260925 12:11:37.491171 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:37.491236 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20260af241334fd5bd6972464b8d1e55, State: Initialized, Role: FOLLOWER
I20260925 12:11:37.491416 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } }
I20260925 12:11:37.491499 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:37.491524 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:37.491552 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:37.492091 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } }
I20260925 12:11:37.492208 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20260af241334fd5bd6972464b8d1e55; no voters:
I20260925 12:11:37.492403 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:37.492432 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:37.492554 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [term 1 LEADER]: Becoming Leader. State: Replica: 20260af241334fd5bd6972464b8d1e55, State: Running, Role: LEADER
I20260925 12:11:37.492635 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } }
I20260925 12:11:37.492697 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:37.492983 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20260af241334fd5bd6972464b8d1e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } } }
I20260925 12:11:37.493162 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:37.493359 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20260af241334fd5bd6972464b8d1e55. Latest consensus state: current_term: 1 leader_uuid: "20260af241334fd5bd6972464b8d1e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20260af241334fd5bd6972464b8d1e55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44637 } } }
I20260925 12:11:37.493434 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55 [sys.catalog]: This master's current role is: LEADER
W20260925 12:11:37.494401 1575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:11:37.494454 1575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:11:37.494498 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:37.494632 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:37.496137 1568 catalog_manager.cc:1347] Generated new cluster ID: 32782caaa21f4fa99328ce3451d5ff30
I20260925 12:11:37.496196 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:37.502177 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.489569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.98.62:44637)
0925 12:11:37.489727 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:37.489735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:37.489772 (+ 37us) server_negotiation.cc:407] Connection header received
0925 12:11:37.490989 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:37.490998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:37.491059 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:37.491229 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:37.491957 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.493274 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.494301 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.494914 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.495045 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.497778 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:37.497782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:37.497782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:37.497793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:37.500303 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.500724 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.500728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.500728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.500834 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.501000 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.501000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.501001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.501305 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:37.501444 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:37.501828 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 12:11:37.501960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260925 12:11:37.507304 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:37.508008 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:37.520941 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Generated new TSK 0
I20260925 12:11:37.521199 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:37.551789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44637
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:37.633513 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:37.633723 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:37.633749 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:37.634923 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:37.634965 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:37.635015 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:37.635052 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:37.636664 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:37.636905 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:37.637138 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:37.642081 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.642086 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.642160 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:37.642320 1580 server_base.cc:1034] running on GCE node
I20260925 12:11:37.642539 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:37.642798 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:37.643978 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790338297643959 us; error 40 us; skew 500 ppm
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:37.646714 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:37.647181 1580 webserver.cc:460] Webserver started at http://127.0.98.1:42287/ using document root <none> and password file <none>
I20260925 12:11:37.647409 1580 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:37.647477 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:37.647634 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:37.648772 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "f45db3adfcc74cfcbb2967c07feb648f"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a1e3647ad349085f082efb648e7bc9da"
server_key_iv: "161c73a0bca8bbd074b5bb322a8fb141"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.649137 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f45db3adfcc74cfcbb2967c07feb648f"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a1e3647ad349085f082efb648e7bc9da"
server_key_iv: "161c73a0bca8bbd074b5bb322a8fb141"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.650523 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:37.651373 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.651587 1580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:37.651705 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "f45db3adfcc74cfcbb2967c07feb648f"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a1e3647ad349085f082efb648e7bc9da"
server_key_iv: "161c73a0bca8bbd074b5bb322a8fb141"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.651780 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:37.665737 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:37.667719 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:37.667884 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:37.668390 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:37.668457 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.668496 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:37.668577 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:37.675668 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33215 every 8 connection(s)
I20260925 12:11:37.675587 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33215
I20260925 12:11:37.676213 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:37.676649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1580
I20260925 12:11:37.676748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:37.676951 392 external_mini_cluster.cc:1428] Setting key 8bc94e50f96322752204d14ea451e3f0
I20260925 12:11:37.678947 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.669031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39485 (local address 127.0.98.62:44637)
0925 12:11:37.669084 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:37.669088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:37.669283 (+ 195us) server_negotiation.cc:407] Connection header received
0925 12:11:37.669929 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:37.669931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:37.669946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:37.670022 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:37.670770 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.671571 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.673007 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.673237 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.673333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.675881 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:37.675883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:37.675883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:37.675891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:37.677526 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.677998 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.677999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.678000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.678062 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.678293 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.678294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.678294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.678435 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:37.678508 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:37.678720 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 12:11:37.678814 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:11:37.679083 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.668978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.1:39485)
0925 12:11:37.669152 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:37.669198 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:37.669769 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:37.670128 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:37.670132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:37.670435 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:37.670668 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.670673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.671664 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.671665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.672922 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.672924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.673357 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.673358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.673484 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.674013 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:37.674022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:37.675765 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:37.677673 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.677679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.677681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.677924 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.678142 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.678143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.678143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.678230 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.678495 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:37.678498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:37.678603 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 12:11:37.678679 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:11:37.678831 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260925 12:11:37.679526 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44637
I20260925 12:11:37.679610 1711 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:37.679838 1711 heartbeater.cc:511] Master 127.0.98.62:44637 requested a full tablet report, sending...
I20260925 12:11:37.680740 1500 ts_manager.cc:194] Registered new tserver with Master: f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215)
I20260925 12:11:37.681576 1500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:11:37.686790 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.680603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58343 (local address 127.0.98.62:44637)
0925 12:11:37.680661 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:37.680663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:37.680801 (+ 138us) server_negotiation.cc:407] Connection header received
0925 12:11:37.680848 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:37.680849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:37.680860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:37.680919 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:37.681278 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.682058 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.682868 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.683189 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.683257 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.684067 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:37.684068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:37.684068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:37.684076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:37.685642 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.686007 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.686008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.686009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.686077 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.686301 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.686302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.686302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.686396 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:37.686444 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:37.686670 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 12:11:37.686730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:11:37.686939 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.680539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.1:58343)
0925 12:11:37.680729 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:37.680740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:37.680801 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:37.680923 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:37.680924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:37.681110 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:37.681176 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.681178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.682172 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.682173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.682784 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.682786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.683267 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.683268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.683314 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.683714 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:37.683718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:37.683973 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:37.685737 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.685738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.685739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.685928 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.686159 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.686160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.686161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.686220 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.686485 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:37.686486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:37.686540 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 12:11:37.686650 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 12:11:37.686753 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:37.722026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44637
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:37.803261 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:37.803514 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:37.803555 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:37.804736 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:37.804805 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:37.804838 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:37.804876 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:37.806427 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:37.806687 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:37.806926 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:37.811916 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.811923 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.811923 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:37.812165 1720 server_base.cc:1034] running on GCE node
I20260925 12:11:37.812381 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:37.812644 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:37.813812 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790338297813805 us; error 35 us; skew 500 ppm
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:37.816206 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:37.816738 1720 webserver.cc:460] Webserver started at http://127.0.98.2:42397/ using document root <none> and password file <none>
I20260925 12:11:37.816962 1720 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:37.817028 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:37.817183 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:37.818234 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "84ff9d77bd6143b6b775e77c8a12e565"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a48a6c9958a2c08f088b45db09704408"
server_key_iv: "d28e4871b0266ab9510be6c7cfed45f7"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.818599 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84ff9d77bd6143b6b775e77c8a12e565"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a48a6c9958a2c08f088b45db09704408"
server_key_iv: "d28e4871b0266ab9510be6c7cfed45f7"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.819921 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:37.820748 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.820922 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:37.821019 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "84ff9d77bd6143b6b775e77c8a12e565"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "a48a6c9958a2c08f088b45db09704408"
server_key_iv: "d28e4871b0266ab9510be6c7cfed45f7"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.821100 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:37.839887 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:37.842064 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:37.842232 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:37.842820 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:37.842872 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.842904 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:37.842924 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.850054 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37043
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:37.850219 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37043 every 8 connection(s)
I20260925 12:11:37.850464 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:37.853294 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.843341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38243 (local address 127.0.98.62:44637)
0925 12:11:37.843408 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:37.843411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:37.843597 (+ 186us) server_negotiation.cc:407] Connection header received
0925 12:11:37.844074 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:37.844076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:37.844088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:37.844170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:37.845143 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.845849 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.847570 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.847912 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.848013 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.850804 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:37.850805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:37.850806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:37.850813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:37.852082 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.852469 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.852470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.852470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.852578 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.852798 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.852799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.852799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.852906 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:37.852956 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:37.853174 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 12:11:37.853232 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:11:37.853474 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.843307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.2:38243)
0925 12:11:37.843460 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:37.843501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:37.843987 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:37.844270 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:37.844275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:37.844743 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:37.845000 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.845006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.845973 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.845974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.847471 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.847475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.848026 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.848027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.848203 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.848786 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:37.848798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:37.850651 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:37.852175 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.852181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.852183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.852401 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.852639 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.852639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.852640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.852740 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.852963 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:37.852967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:37.853082 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 12:11:37.853164 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:11:37.853282 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260925 12:11:37.854110 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44637
I20260925 12:11:37.854188 1851 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:37.854435 1851 heartbeater.cc:511] Master 127.0.98.62:44637 requested a full tablet report, sending...
I20260925 12:11:37.854851 1500 ts_manager.cc:194] Registered new tserver with Master: 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2:37043)
I20260925 12:11:37.855317 1500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38243
I20260925 12:11:37.857034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1720
I20260925 12:11:37.857133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:37.857309 392 external_mini_cluster.cc:1428] Setting key 8ea046b37288eaa522a16ff1235a6e22
I20260925 12:11:37.860675 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.855249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40585 (local address 127.0.98.62:44637)
0925 12:11:37.855294 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:37.855296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:37.855355 (+ 59us) server_negotiation.cc:407] Connection header received
0925 12:11:37.855444 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:37.855445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:37.855454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:37.855501 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:37.855869 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.856329 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.857086 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.857333 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.857408 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.858312 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:37.858313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:37.858313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:37.858320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:37.859544 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.859889 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.859890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.859891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.859944 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:37.860144 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:37.860145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:37.860145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:37.860282 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:37.860343 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:37.860560 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 12:11:37.860619 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:11:37.860860 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:37.855184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.2:40585)
0925 12:11:37.855301 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:37.855312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:37.855371 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:37.855563 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:37.855563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:37.855728 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:37.855782 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.855784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.856420 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.856421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.857003 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:37.857005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:37.857407 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:37.857408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:37.857476 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:37.857888 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:37.857907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:37.858230 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:37.859649 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.859650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.859650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.859819 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.860018 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:37.860019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:37.860020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:37.860079 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:37.860356 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:37.860357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:37.860405 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 12:11:37.860535 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 12:11:37.860700 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:37.888751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44637
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:37.974507 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:37.974715 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:37.974740 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:37.975826 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:37.975862 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:37.975883 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:37.975899 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:37.977384 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:37.977569 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:37.977782 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:37.982846 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.982846 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:37.982952 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:37.983009 1860 server_base.cc:1034] running on GCE node
I20260925 12:11:37.983331 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:37.983585 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:37.984748 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790338297984739 us; error 44 us; skew 500 ppm
Sep 25 12:11:37 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:37.987196 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:37.987680 1860 webserver.cc:460] Webserver started at http://127.0.98.3:46683/ using document root <none> and password file <none>
I20260925 12:11:37.987860 1860 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:37.987902 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:37.987993 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:37.989061 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "a90acbd0b3794bfdbbb4669918b80378"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "b85d28269412ec38ed735287701fc87c"
server_key_iv: "f19d41c31051802990541fd8c9638c92"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.989367 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a90acbd0b3794bfdbbb4669918b80378"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "b85d28269412ec38ed735287701fc87c"
server_key_iv: "f19d41c31051802990541fd8c9638c92"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.990638 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:37.991405 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:37.991600 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:37.991672 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "a90acbd0b3794bfdbbb4669918b80378"
format_stamp: "Formatted at 2026-09-25 12:11:37 on dist-test-slave-2hqq"
server_key: "b85d28269412ec38ed735287701fc87c"
server_key_iv: "f19d41c31051802990541fd8c9638c92"
server_key_version: "encryptionkey@0"
I20260925 12:11:37.991719 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:38.008939 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:38.011206 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:38.011320 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:38.011778 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:38.011818 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.011845 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:38.011904 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.019331 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41393
I20260925 12:11:38.019398 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41393 every 8 connection(s)
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:38.019714 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:38.023365 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.012336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47591 (local address 127.0.98.62:44637)
0925 12:11:38.012408 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.012410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.012696 (+ 286us) server_negotiation.cc:407] Connection header received
0925 12:11:38.013482 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.013483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.013497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.013561 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.014226 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.014739 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.016532 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.016851 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.016925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.020347 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.020348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.020349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.020358 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.021985 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.022409 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.022411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.022411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.022475 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.022727 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.022728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.022728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.022845 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.022921 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.023193 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.023277 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:11:38.023591 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.012316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.3:47591)
0925 12:11:38.012544 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.012595 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.013308 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.013682 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.013687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.013934 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:38.014125 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.014129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.014868 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.014869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.016400 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.016404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.016966 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.016967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.017127 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.018273 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:38.018284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:38.020230 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:38.022095 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.022102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.022104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.022328 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.022545 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.022545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.022546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.022659 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.022912 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:38.022919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:38.023059 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.023170 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.023325 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260925 12:11:38.023761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 1860
I20260925 12:11:38.023852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:38.024044 392 external_mini_cluster.cc:1428] Setting key 9277020cbe38c612c75978ad5a35e256
I20260925 12:11:38.024659 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44637
I20260925 12:11:38.024794 1991 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:38.024979 1991 heartbeater.cc:511] Master 127.0.98.62:44637 requested a full tablet report, sending...
I20260925 12:11:38.025372 1508 ts_manager.cc:194] Registered new tserver with Master: a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393)
I20260925 12:11:38.025825 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47591
I20260925 12:11:38.030820 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.025204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49425 (local address 127.0.98.62:44637)
0925 12:11:38.025243 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.025246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.025361 (+ 115us) server_negotiation.cc:407] Connection header received
0925 12:11:38.025401 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.025402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.025410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.025472 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.025785 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.026381 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.027096 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.027301 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.027382 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.028296 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.028297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.028298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.028304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.029708 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.030064 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.030065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.030065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.030118 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.030321 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.030322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.030322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.030432 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.030506 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.030659 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.030731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:11:38.030833 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.025148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.98.3:49425)
0925 12:11:38.025284 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.025294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.025355 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.025477 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.025478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.025657 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:38.025705 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.025707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.026475 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.026476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.027012 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.027014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.027383 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.027384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.027450 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.027910 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:38.027915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:38.028193 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:38.029806 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.029808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.029808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.029977 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.030192 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.030193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.030194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.030250 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.030505 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:38.030506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:38.030549 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.030636 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.030732 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 12:11:38.035833 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:38.058961 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.051240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.98.62:44637)
0925 12:11:38.051326 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.051328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.051337 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:11:38.051362 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.051363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.051372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.051436 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.052006 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.052591 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.053387 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.053635 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.053738 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.056420 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.056422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.056422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.056429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.057792 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.058166 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.058167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.058167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.058219 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.058383 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.058383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.058384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.058489 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.058604 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.058818 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.058893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 12:11:38.060798 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:38.062410 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:38.069113 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.064929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.0.98.3:41393)
0925 12:11:38.065179 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.065185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.065196 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:38.065253 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.065255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.065273 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.065357 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.065614 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066784 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068049 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068655 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068735 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.068818 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.068975 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.069014 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260925 12:11:38.069154 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.065033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33215 (local address 127.0.0.1:39886)
0925 12:11:38.065259 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.065268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.065305 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.065619 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.065620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.065622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.065657 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.065658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066931 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.066932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.067830 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.067833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068819 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.068890 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.068894 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.068927 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.069005 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
I20260925 12:11:38.069221 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.064994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.0.98.2:37043)
0925 12:11:38.065325 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.065330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.065342 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:11:38.065385 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.065387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.065402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.065479 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.065714 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066423 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.067902 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068720 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068976 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.069058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.069085 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.069126 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":71,"threads_started":1}
I20260925 12:11:38.069159 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.064811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37043 (local address 127.0.0.1:39722)
0925 12:11:38.065109 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.065121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.065178 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.065492 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.065493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.065496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.065561 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.065563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066517 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.066518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.067813 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.067815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068818 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.068890 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.068896 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.068927 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.069058 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":88,"thread_start_us":121,"threads_started":1}
I20260925 12:11:38.069458 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.064807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41393 (local address 127.0.0.1:34092)
0925 12:11:38.064986 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.065027 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.065094 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.065443 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.065446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.065448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.065526 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.065528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066890 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.066891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.067964 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.067965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068746 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.068747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.068841 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.068855 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.068957 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.069005 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260925 12:11:38.069551 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.065283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.0.98.1:33215)
0925 12:11:38.065458 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.065462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.065471 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:11:38.065507 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.065509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.065522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.065739 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.065797 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.066833 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.067945 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.068711 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.069204 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.069346 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.069378 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.069445 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260925 12:11:38.070114 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 55eb9bc9efa24e17961313ee543f6924 (DEFAULT_TABLE table=test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.070261 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 55eb9bc9efa24e17961313ee543f6924 (DEFAULT_TABLE table=test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.070261 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 55eb9bc9efa24e17961313ee543f6924 (DEFAULT_TABLE table=test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.070541 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55eb9bc9efa24e17961313ee543f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.070542 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55eb9bc9efa24e17961313ee543f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.070639 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55eb9bc9efa24e17961313ee543f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.073901 2014 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Bootstrap starting.
I20260925 12:11:38.074285 2015 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Bootstrap starting.
I20260925 12:11:38.074627 2016 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Bootstrap starting.
I20260925 12:11:38.074687 2014 tablet_bootstrap.cc:654] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.075012 2015 tablet_bootstrap.cc:654] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.075047 2014 log.cc:824] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:38.075205 2016 tablet_bootstrap.cc:654] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.075341 2015 log.cc:824] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:38.075559 2016 log.cc:824] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:38.075865 2014 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: No bootstrap required, opened a new log
I20260925 12:11:38.075955 2014 ts_tablet_manager.cc:1397] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:38.076092 2015 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: No bootstrap required, opened a new log
I20260925 12:11:38.076154 2015 ts_tablet_manager.cc:1397] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:38.076290 2016 tablet_bootstrap.cc:492] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: No bootstrap required, opened a new log
I20260925 12:11:38.076344 2016 ts_tablet_manager.cc:1397] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:38.077804 2015 raft_consensus.cc:348] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.077931 2015 raft_consensus.cc:374] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.077952 2015 raft_consensus.cc:729] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90acbd0b3794bfdbbb4669918b80378, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.078080 2015 consensus_queue.cc:260] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.078049 2014 raft_consensus.cc:348] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.078208 2014 raft_consensus.cc:374] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.078244 2014 raft_consensus.cc:729] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45db3adfcc74cfcbb2967c07feb648f, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.078364 1991 heartbeater.cc:503] Master 127.0.98.62:44637 was elected leader, sending a full tablet report...
I20260925 12:11:38.078359 2014 consensus_queue.cc:260] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.078382 2015 ts_tablet_manager.cc:1428] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:11:38.078333 2016 raft_consensus.cc:348] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.078464 2016 raft_consensus.cc:374] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.078504 2016 raft_consensus.cc:729] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ff9d77bd6143b6b775e77c8a12e565, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.078615 2014 ts_tablet_manager.cc:1428] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:11:38.078627 2016 consensus_queue.cc:260] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.078685 1711 heartbeater.cc:503] Master 127.0.98.62:44637 was elected leader, sending a full tablet report...
I20260925 12:11:38.078872 2016 ts_tablet_manager.cc:1428] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:11:38.078974 1851 heartbeater.cc:503] Master 127.0.98.62:44637 was elected leader, sending a full tablet report...
W20260925 12:11:38.101058 1852 tablet.cc:2369] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:11:38.176949 1712 tablet.cc:2369] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:38.181452 2022 raft_consensus.cc:482] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:38.181599 2022 raft_consensus.cc:504] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.181991 2022 leader_election.cc:288] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393), f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215)
I20260925 12:11:38.186061 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.182250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44249 (local address 127.0.98.1:33215)
0925 12:11:38.182291 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.182294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.182451 (+ 157us) server_negotiation.cc:407] Connection header received
0925 12:11:38.182482 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.182484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.182494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.182547 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.182710 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.183479 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185057 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.185598 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185680 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.185690 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.185943 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.185985 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:11:38.186050 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.182331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36435 (local address 127.0.98.3:41393)
0925 12:11:38.182383 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.182385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.182396 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:38.182445 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.182446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.182458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.182523 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.182710 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.183450 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185056 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.185695 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185756 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.185771 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.185943 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.185982 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:11:38.186175 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.182182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41393 (local address 127.0.98.2:36435)
0925 12:11:38.182239 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.182250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.182295 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.182552 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.182553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.182554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.182632 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.182634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.183569 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.183570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.184921 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.184925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.185777 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.185846 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.185853 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.185922 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.185967 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":74}
I20260925 12:11:38.186175 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.182195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33215 (local address 127.0.98.2:44249)
0925 12:11:38.182366 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.182375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.182438 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.182552 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.182553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.182554 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.182639 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.182640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.183569 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.183570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.184921 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.184925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.185713 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.185714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.185846 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.185854 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.185922 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.185967 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":80,"thread_start_us":124,"threads_started":1}
I20260925 12:11:38.186551 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55eb9bc9efa24e17961313ee543f6924" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90acbd0b3794bfdbbb4669918b80378" is_pre_election: true
I20260925 12:11:38.186709 1945 raft_consensus.cc:2454] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 0.
I20260925 12:11:38.186695 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55eb9bc9efa24e17961313ee543f6924" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f" is_pre_election: true
I20260925 12:11:38.186846 1665 raft_consensus.cc:2454] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 0.
I20260925 12:11:38.187275 1737 leader_election.cc:302] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ff9d77bd6143b6b775e77c8a12e565, a90acbd0b3794bfdbbb4669918b80378; no voters:
I20260925 12:11:38.187470 2022 raft_consensus.cc:2781] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:38.187556 2022 raft_consensus.cc:482] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:38.187593 2022 raft_consensus.cc:3037] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.188383 2022 raft_consensus.cc:504] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.188568 2022 leader_election.cc:288] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 election: Requested vote from peers a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393), f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215)
I20260925 12:11:38.188731 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55eb9bc9efa24e17961313ee543f6924" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90acbd0b3794bfdbbb4669918b80378"
I20260925 12:11:38.188817 1945 raft_consensus.cc:3037] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.189473 1945 raft_consensus.cc:2454] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 1.
I20260925 12:11:38.189605 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55eb9bc9efa24e17961313ee543f6924" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f"
I20260925 12:11:38.189704 1737 leader_election.cc:302] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ff9d77bd6143b6b775e77c8a12e565, a90acbd0b3794bfdbbb4669918b80378; no voters:
I20260925 12:11:38.189706 1665 raft_consensus.cc:3037] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.189785 2022 raft_consensus.cc:2781] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:38.189926 2022 raft_consensus.cc:686] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 LEADER]: Becoming Leader. State: Replica: 84ff9d77bd6143b6b775e77c8a12e565, State: Running, Role: LEADER
I20260925 12:11:38.190029 2022 consensus_queue.cc:237] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } }
I20260925 12:11:38.190560 1665 raft_consensus.cc:2454] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 1.
I20260925 12:11:38.190845 1500 catalog_manager.cc:5495] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84ff9d77bd6143b6b775e77c8a12e565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } health_report { overall_health: UNKNOWN } } }
I20260925 12:11:38.214851 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.212426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.0.98.2:37043)
0925 12:11:38.212554 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.212556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.212570 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:11:38.212604 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.212605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.212623 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.212687 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:38.212827 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.213296 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.214051 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.214232 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.214289 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.214519 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.214621 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.214691 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.214770 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.214788 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 12:11:38.216746 1665 raft_consensus.cc:1261] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Refusing update from remote peer 84ff9d77bd6143b6b775e77c8a12e565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.216746 1945 raft_consensus.cc:1261] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Refusing update from remote peer 84ff9d77bd6143b6b775e77c8a12e565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.217104 2024 consensus_queue.cc:1035] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.217252 2022 consensus_queue.cc:1035] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.219692 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333225669492768768 to 7333225669386997760. Current Snapshot: MvccSnapshot[applied={T|T < 7333225669492768768}]
I20260925 12:11:38.219866 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333225669492768768 to 7333225669386997760. Current Snapshot: MvccSnapshot[applied={T|T < 7333225669492768768}]
I20260925 12:11:38.223138 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:38.223279 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:38.224345 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Sending DeleteTablet for 3 replicas of tablet 55eb9bc9efa24e17961313ee543f6924
I20260925 12:11:38.224692 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 55eb9bc9efa24e17961313ee543f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34092
I20260925 12:11:38.224725 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 55eb9bc9efa24e17961313ee543f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39886
I20260925 12:11:38.224896 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 55eb9bc9efa24e17961313ee543f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39722
I20260925 12:11:38.225143 2038 tablet_replica.cc:330] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: stopping tablet replica
I20260925 12:11:38.225157 2037 tablet_replica.cc:330] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: stopping tablet replica
I20260925 12:11:38.225133 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 12:11:38.225251 2037 raft_consensus.cc:2229] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 12:11:38.225265 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:38.225356 2037 raft_consensus.cc:2258] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.225463 2038 raft_consensus.cc:2229] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:38.225467 2039 tablet_replica.cc:330] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: stopping tablet replica
I20260925 12:11:38.225550 2039 raft_consensus.cc:2229] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:38.225657 2039 raft_consensus.cc:2258] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.225831 2038 raft_consensus.cc:2258] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.226122 2039 ts_tablet_manager.cc:1905] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.226280 2038 ts_tablet_manager.cc:1905] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.227005 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 98e079d8e58f4b449b3f258187fda211 (DEFAULT_TABLE table=test-table [id=ce92633bc3614d0098657d9d41663061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.227174 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e079d8e58f4b449b3f258187fda211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.226801 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 98e079d8e58f4b449b3f258187fda211 (DEFAULT_TABLE table=test-table [id=ce92633bc3614d0098657d9d41663061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.227298 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e079d8e58f4b449b3f258187fda211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.228514 2016 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Bootstrap starting.
I20260925 12:11:38.228467 2015 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Bootstrap starting.
I20260925 12:11:38.228482 2038 ts_tablet_manager.cc:1918] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:38.228605 2038 log.cc:1196] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/55eb9bc9efa24e17961313ee543f6924
I20260925 12:11:38.228884 2038 ts_tablet_manager.cc:1939] T 55eb9bc9efa24e17961313ee543f6924 P a90acbd0b3794bfdbbb4669918b80378: Deleting consensus metadata
I20260925 12:11:38.228914 2016 tablet_bootstrap.cc:654] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.228925 2015 tablet_bootstrap.cc:654] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.226886 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 98e079d8e58f4b449b3f258187fda211 (DEFAULT_TABLE table=test-table [id=ce92633bc3614d0098657d9d41663061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:38.229305 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e079d8e58f4b449b3f258187fda211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.229437 2039 ts_tablet_manager.cc:1918] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:38.229496 2039 log.cc:1196] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/55eb9bc9efa24e17961313ee543f6924
I20260925 12:11:38.229553 2016 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: No bootstrap required, opened a new log
I20260925 12:11:38.229599 2016 ts_tablet_manager.cc:1397] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:38.229696 2037 ts_tablet_manager.cc:1905] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.229730 2016 raft_consensus.cc:348] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.229820 2016 raft_consensus.cc:374] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.229838 2016 raft_consensus.cc:729] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ff9d77bd6143b6b775e77c8a12e565, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.229888 2016 consensus_queue.cc:260] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.229980 2016 ts_tablet_manager.cc:1428] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.229732 2039 ts_tablet_manager.cc:1939] T 55eb9bc9efa24e17961313ee543f6924 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting consensus metadata
I20260925 12:11:38.230420 1492 catalog_manager.cc:4841] TS 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2:37043): tablet 55eb9bc9efa24e17961313ee543f6924 (table test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]) successfully deleted
I20260925 12:11:38.230840 2014 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Bootstrap starting.
I20260925 12:11:38.231184 2015 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: No bootstrap required, opened a new log
I20260925 12:11:38.231237 2014 tablet_bootstrap.cc:654] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.231240 2015 ts_tablet_manager.cc:1397] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 12:11:38.231401 2015 raft_consensus.cc:348] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.231518 2015 raft_consensus.cc:374] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.231580 2015 raft_consensus.cc:729] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90acbd0b3794bfdbbb4669918b80378, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.231629 2015 consensus_queue.cc:260] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.231721 2015 ts_tablet_manager.cc:1428] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:38.231823 2014 tablet_bootstrap.cc:492] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: No bootstrap required, opened a new log
I20260925 12:11:38.231820 1493 catalog_manager.cc:4841] TS a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393): tablet 55eb9bc9efa24e17961313ee543f6924 (table test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]) successfully deleted
I20260925 12:11:38.231866 2014 ts_tablet_manager.cc:1397] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:38.231993 2037 ts_tablet_manager.cc:1918] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:38.232051 2037 log.cc:1196] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/55eb9bc9efa24e17961313ee543f6924
I20260925 12:11:38.232019 2014 raft_consensus.cc:348] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.232103 2014 raft_consensus.cc:374] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.232125 2014 raft_consensus.cc:729] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45db3adfcc74cfcbb2967c07feb648f, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.232177 2014 consensus_queue.cc:260] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.232270 2014 ts_tablet_manager.cc:1428] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.232673 2037 ts_tablet_manager.cc:1939] T 55eb9bc9efa24e17961313ee543f6924 P f45db3adfcc74cfcbb2967c07feb648f: Deleting consensus metadata
I20260925 12:11:38.233052 1492 catalog_manager.cc:4841] TS f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215): tablet 55eb9bc9efa24e17961313ee543f6924 (table test-table [id=1b1cf513fe9c435a8ee806e4cb474db3]) successfully deleted
W20260925 12:11:38.270352 1992 tablet.cc:2369] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:38.430440 2034 raft_consensus.cc:482] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:38.430552 2034 raft_consensus.cc:504] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.430895 2034 leader_election.cc:288] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215), a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393)
I20260925 12:11:38.431032 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e079d8e58f4b449b3f258187fda211" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f" is_pre_election: true
I20260925 12:11:38.431166 1665 raft_consensus.cc:2454] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 0.
I20260925 12:11:38.431109 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e079d8e58f4b449b3f258187fda211" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90acbd0b3794bfdbbb4669918b80378" is_pre_election: true
I20260925 12:11:38.431247 1945 raft_consensus.cc:2454] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 0.
I20260925 12:11:38.431360 1736 leader_election.cc:302] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ff9d77bd6143b6b775e77c8a12e565, f45db3adfcc74cfcbb2967c07feb648f; no voters:
I20260925 12:11:38.431457 2034 raft_consensus.cc:2781] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:38.431504 2034 raft_consensus.cc:482] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:38.431523 2034 raft_consensus.cc:3037] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.432240 2034 raft_consensus.cc:504] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.432371 2034 leader_election.cc:288] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 election: Requested vote from peers f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215), a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393)
I20260925 12:11:38.432512 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e079d8e58f4b449b3f258187fda211" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f"
I20260925 12:11:38.432601 1665 raft_consensus.cc:3037] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.432582 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e079d8e58f4b449b3f258187fda211" candidate_uuid: "84ff9d77bd6143b6b775e77c8a12e565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90acbd0b3794bfdbbb4669918b80378"
I20260925 12:11:38.432651 1945 raft_consensus.cc:3037] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.433395 1945 raft_consensus.cc:2454] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 1.
I20260925 12:11:38.433395 1665 raft_consensus.cc:2454] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84ff9d77bd6143b6b775e77c8a12e565 in term 1.
I20260925 12:11:38.433705 1736 leader_election.cc:302] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ff9d77bd6143b6b775e77c8a12e565, f45db3adfcc74cfcbb2967c07feb648f; no voters:
I20260925 12:11:38.433840 2034 raft_consensus.cc:2781] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:38.433889 2034 raft_consensus.cc:686] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 LEADER]: Becoming Leader. State: Replica: 84ff9d77bd6143b6b775e77c8a12e565, State: Running, Role: LEADER
I20260925 12:11:38.433925 2034 consensus_queue.cc:237] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.434634 1508 catalog_manager.cc:5495] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84ff9d77bd6143b6b775e77c8a12e565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } health_report { overall_health: UNKNOWN } } }
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:38.488268 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.477405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.0.98.62:44637)
0925 12:11:38.477477 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.477479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.477580 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:11:38.477630 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.477630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.477639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.477696 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.478143 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.478660 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.479338 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.479564 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.479621 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.485646 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.485648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.485648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.485654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.487079 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.487530 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.487531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.487531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.487581 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.487789 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.487789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.487789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.487898 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.487948 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.488128 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.488200 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:11:38.488433 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.477340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44637 (local address 127.0.0.1:37262)
0925 12:11:38.477514 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.477524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.477584 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.477716 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.477717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.478020 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:38.478069 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.478071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.478732 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.478733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.479255 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.479256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.479648 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.479649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.479700 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.484045 (+ 4345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:38.484050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:38.485516 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:38.487175 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.487182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.487184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.487397 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.487671 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:38.487671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:38.487672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:38.487732 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:38.487976 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:38.487984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:38.488035 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.488108 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.488283 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":67,"thread_start_us":92,"threads_started":1}
I20260925 12:11:38.490485 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37262:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:11:38.490710 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:38.492305 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 079e9695ba254a12b54466cd8b3659e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1adf2b606f40a0b5625271baa072c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:38.492338 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 079e9695ba254a12b54466cd8b3659e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1adf2b606f40a0b5625271baa072c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:38.492475 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079e9695ba254a12b54466cd8b3659e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.492623 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 079e9695ba254a12b54466cd8b3659e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e1adf2b606f40a0b5625271baa072c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:38.492748 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079e9695ba254a12b54466cd8b3659e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.494143 2015 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: Bootstrap starting.
I20260925 12:11:38.494259 2016 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: Bootstrap starting.
I20260925 12:11:38.492476 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079e9695ba254a12b54466cd8b3659e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:38.494603 2015 tablet_bootstrap.cc:654] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.494627 2016 tablet_bootstrap.cc:654] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.495635 2015 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: No bootstrap required, opened a new log
I20260925 12:11:38.495700 2015 ts_tablet_manager.cc:1397] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:11:38.495857 2015 raft_consensus.cc:348] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.495934 2015 raft_consensus.cc:374] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.495962 2015 raft_consensus.cc:729] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90acbd0b3794bfdbbb4669918b80378, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.495976 2014 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: Bootstrap starting.
I20260925 12:11:38.496001 2015 consensus_queue.cc:260] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.496091 2015 ts_tablet_manager.cc:1428] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.496153 2020 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.496237 2020 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.496313 2016 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: No bootstrap required, opened a new log
I20260925 12:11:38.496361 2016 ts_tablet_manager.cc:1397] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:38.496395 2014 tablet_bootstrap.cc:654] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:38.496470 2016 raft_consensus.cc:348] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.496577 2016 raft_consensus.cc:374] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.496604 2016 raft_consensus.cc:729] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ff9d77bd6143b6b775e77c8a12e565, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.496649 2016 consensus_queue.cc:260] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.496749 2016 ts_tablet_manager.cc:1428] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.496851 2034 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.496925 2034 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.497113 2014 tablet_bootstrap.cc:492] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: No bootstrap required, opened a new log
I20260925 12:11:38.497170 2014 ts_tablet_manager.cc:1397] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:38.497327 2014 raft_consensus.cc:348] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.497388 2014 raft_consensus.cc:374] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:38.497402 2014 raft_consensus.cc:729] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45db3adfcc74cfcbb2967c07feb648f, State: Initialized, Role: FOLLOWER
I20260925 12:11:38.497463 2014 consensus_queue.cc:260] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.497553 2014 ts_tablet_manager.cc:1428] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:38.497686 2021 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.497758 2021 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.598490 2020 raft_consensus.cc:482] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:38.598620 2020 raft_consensus.cc:504] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.599011 2020 leader_election.cc:288] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215), 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2:37043)
I20260925 12:11:38.604305 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.599587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37043 (local address 127.0.98.3:58215)
0925 12:11:38.599759 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.599769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.599858 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.600149 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.600149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.600152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.600256 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.600260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.601497 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.601498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.603129 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.603132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.604030 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.604031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.604106 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.604111 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.604163 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.604207 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":50,"threads_started":1}
I20260925 12:11:38.604332 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.599679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58215 (local address 127.0.98.2:37043)
0925 12:11:38.599736 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.599738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.600031 (+ 293us) server_negotiation.cc:407] Connection header received
0925 12:11:38.600060 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.600061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.600069 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.600129 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.600340 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.601390 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.603268 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.603890 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.603988 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.604001 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.604197 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.604247 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:11:38.604992 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.599781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51197 (local address 127.0.98.1:33215)
0925 12:11:38.599823 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.599825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.599834 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:11:38.599865 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.599866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.599874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.600194 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:38.601559 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.602433 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.603858 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.604441 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.604576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.604588 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.604823 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.604864 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:11:38.604785 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079e9695ba254a12b54466cd8b3659e0" candidate_uuid: "a90acbd0b3794bfdbbb4669918b80378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ff9d77bd6143b6b775e77c8a12e565" is_pre_election: true
I20260925 12:11:38.605116 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.599246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33215 (local address 127.0.98.3:51197)
0925 12:11:38.599382 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.599421 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.599517 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.600233 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.600233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.600235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:38.600285 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.600287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.602535 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.602536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.603773 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.603775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.604632 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.604633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.604730 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.604735 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.604792 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.604846 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260925 12:11:38.605111 1805 raft_consensus.cc:2454] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90acbd0b3794bfdbbb4669918b80378 in term 0.
I20260925 12:11:38.605413 1876 leader_election.cc:302] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84ff9d77bd6143b6b775e77c8a12e565, a90acbd0b3794bfdbbb4669918b80378; no voters:
I20260925 12:11:38.605572 2020 raft_consensus.cc:2781] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:38.605615 2020 raft_consensus.cc:482] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:38.605638 2020 raft_consensus.cc:3037] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.605755 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079e9695ba254a12b54466cd8b3659e0" candidate_uuid: "a90acbd0b3794bfdbbb4669918b80378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f" is_pre_election: true
I20260925 12:11:38.605849 1665 raft_consensus.cc:2454] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90acbd0b3794bfdbbb4669918b80378 in term 0.
I20260925 12:11:38.606666 2020 raft_consensus.cc:504] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.606825 2020 leader_election.cc:288] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [CANDIDATE]: Term 1 election: Requested vote from peers f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215), 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2:37043)
I20260925 12:11:38.607012 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079e9695ba254a12b54466cd8b3659e0" candidate_uuid: "a90acbd0b3794bfdbbb4669918b80378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45db3adfcc74cfcbb2967c07feb648f"
I20260925 12:11:38.607096 1665 raft_consensus.cc:3037] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.607034 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079e9695ba254a12b54466cd8b3659e0" candidate_uuid: "a90acbd0b3794bfdbbb4669918b80378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ff9d77bd6143b6b775e77c8a12e565"
I20260925 12:11:38.607136 1805 raft_consensus.cc:3037] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:38.607707 1665 raft_consensus.cc:2454] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90acbd0b3794bfdbbb4669918b80378 in term 1.
I20260925 12:11:38.607898 1876 leader_election.cc:302] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a90acbd0b3794bfdbbb4669918b80378, f45db3adfcc74cfcbb2967c07feb648f; no voters:
I20260925 12:11:38.607992 2020 raft_consensus.cc:2781] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:38.607986 1805 raft_consensus.cc:2454] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90acbd0b3794bfdbbb4669918b80378 in term 1.
I20260925 12:11:38.608091 2020 raft_consensus.cc:686] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [term 1 LEADER]: Becoming Leader. State: Replica: a90acbd0b3794bfdbbb4669918b80378, State: Running, Role: LEADER
I20260925 12:11:38.608165 2020 consensus_queue.cc:237] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } }
I20260925 12:11:38.608439 2055 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: TxnStatusTablet state changed. Reason: New leader a90acbd0b3794bfdbbb4669918b80378. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.608539 2055 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:38.608753 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:11:38.608922 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:11:38.608863 1500 catalog_manager.cc:5495] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 reported cstate change: term changed from 0 to 1, leader changed from <none> to a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } health_report { overall_health: HEALTHY } } }
I20260925 12:11:38.643726 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.640169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.98.3:41393)
0925 12:11:38.640214 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.640216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.640247 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:11:38.640354 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.640355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.640377 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.640482 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:38.640664 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.641433 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.642669 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.643006 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.643081 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.643307 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.643438 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.643514 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.643610 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.643632 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:11:38.643769 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.640107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41393 (local address 127.0.0.1:34094)
0925 12:11:38.640195 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.640207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.640268 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.640515 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.640517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.640518 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:38.640582 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.640584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.641517 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.641518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.642567 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.642570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.643080 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.643081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.643176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.643199 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.643516 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.643540 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.643605 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.643665 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 12:11:38.644984 1665 raft_consensus.cc:1261] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Refusing update from remote peer a90acbd0b3794bfdbbb4669918b80378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.645112 1805 raft_consensus.cc:1261] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Refusing update from remote peer a90acbd0b3794bfdbbb4669918b80378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.645370 2020 consensus_queue.cc:1035] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.645533 2055 consensus_queue.cc:1035] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.646181 2021 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: TxnStatusTablet state changed. Reason: New leader a90acbd0b3794bfdbbb4669918b80378. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.646281 2021 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.646230 2034 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: TxnStatusTablet state changed. Reason: New leader a90acbd0b3794bfdbbb4669918b80378. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.646317 2034 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.647452 2020 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.647543 2020 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:38.647614 2020 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.647662 2020 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P a90acbd0b3794bfdbbb4669918b80378: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:38.648021 2021 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.648118 2021 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P f45db3adfcc74cfcbb2967c07feb648f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.648141 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333225671247732736 to 7333225671099695104. Current Snapshot: MvccSnapshot[applied={T|T < 7333225671247732736}]
I20260925 12:11:38.648231 2034 tablet_replica.cc:439] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a90acbd0b3794bfdbbb4669918b80378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 } } peers { permanent_uuid: "84ff9d77bd6143b6b775e77c8a12e565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37043 } } peers { permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 } } }
I20260925 12:11:38.648293 2034 tablet_replica.cc:442] T 079e9695ba254a12b54466cd8b3659e0 P 84ff9d77bd6143b6b775e77c8a12e565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:38.655791 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.652448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41393 (local address 127.0.98.2:46021)
0925 12:11:38.652696 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.652706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.652788 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.652946 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.652947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.652949 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:38.653013 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.653016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.653977 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.653978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.654742 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.654744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.655207 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.655208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.655311 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.655327 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.655617 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.655619 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.655673 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.655720 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
I20260925 12:11:38.655772 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.652514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46021 (local address 127.0.98.3:41393)
0925 12:11:38.652557 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.652560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.652771 (+ 211us) server_negotiation.cc:407] Connection header received
0925 12:11:38.652839 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.652840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.652850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.652900 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:38.653093 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.653859 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.654839 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.655134 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.655208 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.655407 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.655456 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.655613 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.655679 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.655697 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:38.657955 1665 raft_consensus.cc:1261] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Refusing update from remote peer 84ff9d77bd6143b6b775e77c8a12e565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.657946 1945 raft_consensus.cc:1261] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Refusing update from remote peer 84ff9d77bd6143b6b775e77c8a12e565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:38.658175 2034 consensus_queue.cc:1035] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90acbd0b3794bfdbbb4669918b80378" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.658267 2024 consensus_queue.cc:1035] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45db3adfcc74cfcbb2967c07feb648f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:38.667429 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.664068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37043 (local address 127.0.98.3:37305)
0925 12:11:38.664247 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:38.664263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:38.664342 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:38.664622 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:38.664622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:38.664623 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:38.664677 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.664679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.665609 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.665610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.666575 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:38.666577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.667040 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.667040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:38.667086 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.667100 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.667276 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.667277 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:38.667312 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 12:11:38.667347 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 12:11:38.667421 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.664121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37305 (local address 127.0.98.2:37043)
0925 12:11:38.664221 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.664224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.664331 (+ 107us) server_negotiation.cc:407] Connection header received
0925 12:11:38.664429 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.664430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.664444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.664526 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:38.664746 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.665492 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.666650 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.666954 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.667027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.667179 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:38.667216 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:38.667270 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.667316 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.667338 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 12:11:38.676365 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:38.676474 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:38.677132 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20260af241334fd5bd6972464b8d1e55: Sending DeleteTablet for 3 replicas of tablet 98e079d8e58f4b449b3f258187fda211
I20260925 12:11:38.677394 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e079d8e58f4b449b3f258187fda211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39886
I20260925 12:11:38.677548 2037 tablet_replica.cc:330] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: stopping tablet replica
I20260925 12:11:38.677591 2037 raft_consensus.cc:2229] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:38.677621 2037 raft_consensus.cc:2258] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.677940 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e079d8e58f4b449b3f258187fda211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34092
I20260925 12:11:38.678027 2038 tablet_replica.cc:330] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: stopping tablet replica
I20260925 12:11:38.678071 2038 raft_consensus.cc:2229] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:38.678102 2038 raft_consensus.cc:2258] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.678357 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e079d8e58f4b449b3f258187fda211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39722
I20260925 12:11:38.678445 2039 tablet_replica.cc:330] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: stopping tablet replica
I20260925 12:11:38.678493 2039 raft_consensus.cc:2229] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:38.678555 2039 raft_consensus.cc:2258] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:38.678903 2039 ts_tablet_manager.cc:1905] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.680773 2037 ts_tablet_manager.cc:1905] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.680807 2039 ts_tablet_manager.cc:1918] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:38.680869 2039 log.cc:1196] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/98e079d8e58f4b449b3f258187fda211
I20260925 12:11:38.681106 2039 ts_tablet_manager.cc:1939] T 98e079d8e58f4b449b3f258187fda211 P 84ff9d77bd6143b6b775e77c8a12e565: Deleting consensus metadata
I20260925 12:11:38.681432 1492 catalog_manager.cc:4841] TS 84ff9d77bd6143b6b775e77c8a12e565 (127.0.98.2:37043): tablet 98e079d8e58f4b449b3f258187fda211 (table test-table [id=ce92633bc3614d0098657d9d41663061]) successfully deleted
I20260925 12:11:38.682269 2037 ts_tablet_manager.cc:1918] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:38.682334 2037 log.cc:1196] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/98e079d8e58f4b449b3f258187fda211
I20260925 12:11:38.682546 2037 ts_tablet_manager.cc:1939] T 98e079d8e58f4b449b3f258187fda211 P f45db3adfcc74cfcbb2967c07feb648f: Deleting consensus metadata
I20260925 12:11:38.682973 2038 ts_tablet_manager.cc:1905] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:38.684479 2038 ts_tablet_manager.cc:1918] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:38.684579 2038 log.cc:1196] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/98e079d8e58f4b449b3f258187fda211
I20260925 12:11:38.684798 2038 ts_tablet_manager.cc:1939] T 98e079d8e58f4b449b3f258187fda211 P a90acbd0b3794bfdbbb4669918b80378: Deleting consensus metadata
I20260925 12:11:38.684868 1492 catalog_manager.cc:4841] TS f45db3adfcc74cfcbb2967c07feb648f (127.0.98.1:33215): tablet 98e079d8e58f4b449b3f258187fda211 (table test-table [id=ce92633bc3614d0098657d9d41663061]) successfully deleted
I20260925 12:11:38.685129 1493 catalog_manager.cc:4841] TS a90acbd0b3794bfdbbb4669918b80378 (127.0.98.3:41393): tablet 98e079d8e58f4b449b3f258187fda211 (table test-table [id=ce92633bc3614d0098657d9d41663061]) successfully deleted
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:38.690596 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.678154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.98.1:33215)
0925 12:11:38.678193 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.678195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.680823 (+ 2628us) server_negotiation.cc:407] Connection header received
0925 12:11:38.681131 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.681133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.681147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.681179 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.681983 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.682798 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.683770 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.684081 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.684159 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.687405 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.687416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.687416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.687424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.689300 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.689682 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.689686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.689686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.689767 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.690002 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.690003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.690003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.690178 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.690323 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.690390 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.690475 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:11:38.690923 1695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39894
I20260925 12:11:38.698868 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:38.691900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.98.62:44637)
0925 12:11:38.691953 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:38.691955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:38.691963 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:38.692042 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:38.692042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:38.692051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:38.692108 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:38.692786 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.693314 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.694417 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:38.694672 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:38.694728 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:38.696326 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:38.696328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:38.696328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:38.696335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:38.697791 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.698176 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.698177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.698178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.698229 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:38.698426 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:38.698427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:38.698427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:38.698529 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:38.698604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:38.698741 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 12:11:38.698803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 12:11:38.699302 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37266
I20260925 12:11:38.699977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1580
I20260925 12:11:38.706897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1720
I20260925 12:11:38.713830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1860
I20260925 12:11:38.719204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 1476
2026-09-25T12:11:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2680 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[2077](info): set up 2 sockets
Sep 25 12:11:38 dist-test-slave-2hqq krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260925 12:11:39.752252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:39 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:39.883659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:39.966475 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:39.966682 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:39.966706 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:39.967803 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:39.967841 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:11:39.967854 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:39.967867 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:39.967880 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:39.967931 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:39.969574 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35259
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:39.969769 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:39.969941 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:39.974697 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:39.974709 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:39.974972 2090 server_base.cc:1034] running on GCE node
W20260925 12:11:39.974709 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:39.975617 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:39.975864 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:39.977051 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790338299977031 us; error 41 us; skew 500 ppm
Sep 25 12:11:39 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:39.979514 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:39.979980 2090 webserver.cc:460] Webserver started at http://127.0.98.62:35091/ using document root <none> and password file <none>
I20260925 12:11:39.980201 2090 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:39.980245 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:39.980386 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:39.981513 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "9ef32fbe20c34c3a95b3202c72511516"
format_stamp: "Formatted at 2026-09-25 12:11:39 on dist-test-slave-2hqq"
server_key: "bc07b97e3dc04d45c1e2c91b48d521c2"
server_key_iv: "48ccc3a521024adf1635e11113d1fa30"
server_key_version: "encryptionkey@0"
I20260925 12:11:39.981873 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ef32fbe20c34c3a95b3202c72511516"
format_stamp: "Formatted at 2026-09-25 12:11:39 on dist-test-slave-2hqq"
server_key: "bc07b97e3dc04d45c1e2c91b48d521c2"
server_key_iv: "48ccc3a521024adf1635e11113d1fa30"
server_key_version: "encryptionkey@0"
I20260925 12:11:39.983170 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:39.983867 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:39.984042 2090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:39.984144 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "9ef32fbe20c34c3a95b3202c72511516"
format_stamp: "Formatted at 2026-09-25 12:11:39 on dist-test-slave-2hqq"
server_key: "bc07b97e3dc04d45c1e2c91b48d521c2"
server_key_iv: "48ccc3a521024adf1635e11113d1fa30"
server_key_version: "encryptionkey@0"
I20260925 12:11:39.984222 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:39.996675 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:39.998621 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:39.998790 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:40.003675 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35259
I20260925 12:11:40.003733 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35259 every 8 connection(s)
I20260925 12:11:40.004158 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:40.004892 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.007112 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Bootstrap starting.
I20260925 12:11:40.007658 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.007886 2168 log.cc:824] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:40.008631 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: No bootstrap required, opened a new log
I20260925 12:11:40.008746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2090
I20260925 12:11:40.008854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:40.009030 392 external_mini_cluster.cc:1428] Setting key 962d935417ea676febc8e33162ff0be8
I20260925 12:11:40.011106 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } }
I20260925 12:11:40.011277 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.011317 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef32fbe20c34c3a95b3202c72511516, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.011454 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } }
I20260925 12:11:40.011541 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:40.011590 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:40.011641 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.012385 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } }
I20260925 12:11:40.012475 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef32fbe20c34c3a95b3202c72511516; no voters:
I20260925 12:11:40.012704 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:40.012729 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:40.012837 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [term 1 LEADER]: Becoming Leader. State: Replica: 9ef32fbe20c34c3a95b3202c72511516, State: Running, Role: LEADER
I20260925 12:11:40.012926 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } }
I20260925 12:11:40.012981 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:40.013273 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef32fbe20c34c3a95b3202c72511516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } }
I20260925 12:11:40.013356 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:40.013495 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef32fbe20c34c3a95b3202c72511516. Latest consensus state: current_term: 1 leader_uuid: "9ef32fbe20c34c3a95b3202c72511516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef32fbe20c34c3a95b3202c72511516" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35259 } } }
I20260925 12:11:40.013567 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:40.013890 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:40.014413 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:40.016016 2179 catalog_manager.cc:1347] Generated new cluster ID: 79e9c63455684727ade7c2632fe715c9
I20260925 12:11:40.016067 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.021270 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.009575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.0.98.62:35259)
0925 12:11:40.009753 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.009762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.009798 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:11:40.011089 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.011096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.011147 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.011267 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.012128 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.013386 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.014492 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.014914 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.015040 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.017792 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.017796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.017796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.017808 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.019598 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.019978 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.019982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.019982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.020094 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.020337 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.020338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.020339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.020650 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.020835 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.020914 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.021060 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 12:11:40.050029 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:40.050611 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:40.058871 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Generated new TSK 0
I20260925 12:11:40.059080 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:40.096012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35259
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:40.179061 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:40.179272 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:40.179301 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:40.180572 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:40.180616 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:40.180637 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:11:40.180680 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:40.180712 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:40.182224 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:40.182480 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:40.182713 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:40.187722 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:40.187711 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:40.187834 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:40.187991 2194 server_base.cc:1034] running on GCE node
I20260925 12:11:40.188194 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:40.188444 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:40.189632 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790338300189619 us; error 28 us; skew 500 ppm
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:40.192119 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:40.192662 2194 webserver.cc:460] Webserver started at http://127.0.98.1:43739/ using document root <none> and password file <none>
I20260925 12:11:40.192893 2194 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:40.192963 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:40.193121 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:40.194185 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "30f220ccbab145efb3196809e86e1f23"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "6e4e62a28ed59525546c56dc11942ed9"
server_key_iv: "2f19482f7b45aaf58f7f00cc58041ee0"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.194566 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30f220ccbab145efb3196809e86e1f23"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "6e4e62a28ed59525546c56dc11942ed9"
server_key_iv: "2f19482f7b45aaf58f7f00cc58041ee0"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.195940 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:11:40.196748 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.196915 2194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:40.197027 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "30f220ccbab145efb3196809e86e1f23"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "6e4e62a28ed59525546c56dc11942ed9"
server_key_iv: "2f19482f7b45aaf58f7f00cc58041ee0"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.197105 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:40.231204 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:40.233151 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:40.233325 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:40.233846 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:40.233882 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.233904 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:40.233917 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.240989 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34327
I20260925 12:11:40.241060 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34327 every 8 connection(s)
I20260925 12:11:40.241384 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.245536 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.234480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51435 (local address 127.0.98.62:35259)
0925 12:11:40.234536 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.234538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.234800 (+ 262us) server_negotiation.cc:407] Connection header received
0925 12:11:40.235416 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.235417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.235437 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.235497 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.236470 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.237303 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.239019 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.239216 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.239274 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.242050 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.242051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.242052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.242059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.243677 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.244280 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.244281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.244282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.244350 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.244687 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.244688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.244689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.244867 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.244966 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.245320 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.245442 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:11:40.245757 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.234494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.1:51435)
0925 12:11:40.234678 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.234714 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.235268 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.235625 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.235631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.236040 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.236355 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.236363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.237409 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.237411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.238919 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.238922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.239307 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.239308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.239478 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.240099 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.240111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.241956 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.243788 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.243795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.243798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.244060 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.244441 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.244442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.244442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.244598 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.244974 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.244980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.245145 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.245285 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.245496 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260925 12:11:40.246248 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35259
I20260925 12:11:40.246338 2325 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:40.246537 2325 heartbeater.cc:511] Master 127.0.98.62:35259 requested a full tablet report, sending...
I20260925 12:11:40.247049 2122 ts_manager.cc:194] Registered new tserver with Master: 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327)
I20260925 12:11:40.247962 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51435
I20260925 12:11:40.251508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2194
I20260925 12:11:40.251621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:40.251811 392 external_mini_cluster.cc:1428] Setting key 44644888a4ffbf0f7e467cf63bbe04f3
I20260925 12:11:40.252753 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.247226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43905 (local address 127.0.98.62:35259)
0925 12:11:40.247271 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.247272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.247368 (+ 96us) server_negotiation.cc:407] Connection header received
0925 12:11:40.247481 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.247481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.247490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.247530 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.247854 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.248325 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.249157 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.249382 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.249460 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.250348 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.250350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.250350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.250357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.251560 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.251961 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.251962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.251963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.252022 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.252234 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.252234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.252235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.252342 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.252421 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.252587 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.252666 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:11:40.252774 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.247188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.1:43905)
0925 12:11:40.247312 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.247323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.247384 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.247545 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.247545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.247725 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.247793 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.247794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.248431 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.248432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.249054 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.249056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.249468 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.249469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.249575 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.249962 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.249966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.250260 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.251697 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.251698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.251699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.251902 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.252098 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.252099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.252099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.252163 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.252412 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.252412 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.252468 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.252563 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.252669 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:40.284024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35259
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:40.364795 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:40.365010 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:40.365036 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:40.366199 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:40.366245 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:40.366266 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:11:40.366277 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:40.366324 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:40.367789 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:40.368042 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:40.368301 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:40.373660 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:40.373668 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:40.373759 2334 server_base.cc:1034] running on GCE node
W20260925 12:11:40.373661 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:40.374061 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:40.374327 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:40.375475 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790338300375477 us; error 41 us; skew 500 ppm
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:40.378042 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:40.378558 2334 webserver.cc:460] Webserver started at http://127.0.98.2:41001/ using document root <none> and password file <none>
I20260925 12:11:40.378789 2334 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:40.378855 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:40.378991 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:40.380036 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "68977ac7ce078d76d1546fee8a4a83f7"
server_key_iv: "cb26d3aa7d4413b0803ce9d120eb8199"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.380422 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "68977ac7ce078d76d1546fee8a4a83f7"
server_key_iv: "cb26d3aa7d4413b0803ce9d120eb8199"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.381764 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:40.382506 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.382673 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:40.382776 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "68977ac7ce078d76d1546fee8a4a83f7"
server_key_iv: "cb26d3aa7d4413b0803ce9d120eb8199"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.382850 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:40.401142 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:40.403169 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:40.403347 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:40.403899 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:40.403964 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.404021 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:40.404048 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.411307 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38679
I20260925 12:11:40.411386 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38679 every 8 connection(s)
I20260925 12:11:40.411747 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:40.414458 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.404552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51459 (local address 127.0.98.62:35259)
0925 12:11:40.404602 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.404607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.404899 (+ 292us) server_negotiation.cc:407] Connection header received
0925 12:11:40.405468 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.405470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.405483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.405568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.406541 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.407389 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.408914 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.409134 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.409223 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.411505 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.411506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.411507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.411514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.413212 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.413594 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.413595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.413596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.413675 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.413891 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.413891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.413892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.414002 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.414084 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.414323 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.414388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:40.414637 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.404547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.2:51459)
0925 12:11:40.404757 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.404806 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.405350 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.405692 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.405696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.406093 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.406413 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.406420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.407495 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.407496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.408804 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.408807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.409246 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.409247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.409379 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.409946 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.409955 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.411398 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.413279 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.413284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.413285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.413509 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.413736 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.413736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.413737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.413834 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.414079 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.414084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.414209 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.414294 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.414417 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260925 12:11:40.415148 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35259
I20260925 12:11:40.415232 2465 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:40.415413 2465 heartbeater.cc:511] Master 127.0.98.62:35259 requested a full tablet report, sending...
I20260925 12:11:40.415807 2121 ts_manager.cc:194] Registered new tserver with Master: 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679)
I20260925 12:11:40.416272 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51459
I20260925 12:11:40.419081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2334
I20260925 12:11:40.419152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:40.419344 392 external_mini_cluster.cc:1428] Setting key 42bd50ede42da75cfb7e45c4a060a9dd
I20260925 12:11:40.422220 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.416277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41427 (local address 127.0.98.62:35259)
0925 12:11:40.416316 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.416318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.416535 (+ 217us) server_negotiation.cc:407] Connection header received
0925 12:11:40.416598 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.416599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.416610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.416659 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.417009 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.417450 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.418112 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.418282 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.418329 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.419305 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.419307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.419307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.419314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.420839 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.421277 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.421278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.421279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.421351 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.421629 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.421631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.421631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.421778 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.421872 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.422045 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.422127 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:11:40.422235 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.416222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.2:41427)
0925 12:11:40.416452 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.416463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.416549 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.416674 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.416674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.416865 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.416932 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.416934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.417556 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.417556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.418043 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.418044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.418343 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.418343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.418398 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.418844 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.418848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.419218 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.420918 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.420920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.420920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.421157 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.421480 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.421481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.421481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.421548 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.421862 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.421863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.421921 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.422020 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.422128 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":175,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:40.451476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35259
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:40.532732 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:40.532971 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:40.533015 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:40.534168 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:40.534241 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:40.534279 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 12:11:40.534325 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:11:40.534365 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:40.535851 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:40.536139 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:40.536387 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:40.541613 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:40.541622 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:40.541752 2474 server_base.cc:1034] running on GCE node
W20260925 12:11:40.541921 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:40.542156 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:40.542400 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:40.543578 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790338300543560 us; error 28 us; skew 500 ppm
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:40.546233 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:40.546747 2474 webserver.cc:460] Webserver started at http://127.0.98.3:33735/ using document root <none> and password file <none>
I20260925 12:11:40.547029 2474 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:40.547080 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:40.547225 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:40.548333 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "aac147094ddc4659a338cbb291f21775"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "1e7be117a5dc0687ed589dcdd1c3fe52"
server_key_iv: "f5f25ec7611dee75a732bf26ca74d27f"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.548717 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aac147094ddc4659a338cbb291f21775"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "1e7be117a5dc0687ed589dcdd1c3fe52"
server_key_iv: "f5f25ec7611dee75a732bf26ca74d27f"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.549957 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:40.550627 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.550810 2474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:40.550915 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "aac147094ddc4659a338cbb291f21775"
format_stamp: "Formatted at 2026-09-25 12:11:40 on dist-test-slave-2hqq"
server_key: "1e7be117a5dc0687ed589dcdd1c3fe52"
server_key_iv: "f5f25ec7611dee75a732bf26ca74d27f"
server_key_version: "encryptionkey@0"
I20260925 12:11:40.551003 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:40.562767 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:40.564785 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:40.564954 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:40.565481 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:40.565547 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.565591 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:40.565635 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.572243 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37947
I20260925 12:11:40.572304 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37947 every 8 connection(s)
I20260925 12:11:40.572685 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.575665 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.565972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33625 (local address 127.0.98.62:35259)
0925 12:11:40.566039 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.566042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.566263 (+ 221us) server_negotiation.cc:407] Connection header received
0925 12:11:40.566749 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.566750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.566763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.566850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.567692 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.568558 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.570192 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.570549 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.570634 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.573317 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.573318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.573319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.573325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.574492 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.574852 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.574853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.574853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.574901 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.575099 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.575100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.575100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.575199 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.575248 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.575519 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.575590 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:11:40.575758 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.565948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.3:33625)
0925 12:11:40.566120 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.566161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.566650 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.566901 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.566904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.567283 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.567583 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.567589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.568671 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.568672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.570090 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.570094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.570633 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.570634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.570735 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.571330 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.571340 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.573237 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.574570 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.574576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.574578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.574788 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.574956 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.574957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.574957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.575049 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.575253 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.575257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.575368 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.575451 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.575571 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260925 12:11:40.576272 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35259
I20260925 12:11:40.576424 2605 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:40.576427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2474
I20260925 12:11:40.576522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:40.576658 2605 heartbeater.cc:511] Master 127.0.98.62:35259 requested a full tablet report, sending...
I20260925 12:11:40.576740 392 external_mini_cluster.cc:1428] Setting key 3451cb3d8ff62cadc772b7e7fbe9d478
I20260925 12:11:40.577049 2121 ts_manager.cc:194] Registered new tserver with Master: aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:40.577555 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:11:40.577574 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33625
I20260925 12:11:40.583823 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.577391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46799 (local address 127.0.98.62:35259)
0925 12:11:40.577433 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.577435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.577566 (+ 131us) server_negotiation.cc:407] Connection header received
0925 12:11:40.577625 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.577626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.577636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.577691 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.578163 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.578721 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.579731 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.580038 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.580115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.581158 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.581160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.581161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.581168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.582684 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.583099 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.583100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.583100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.583153 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.583352 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.583353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.583353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.583459 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.583528 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.583680 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.583739 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:11:40.583855 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.577345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.98.3:46799)
0925 12:11:40.577499 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.577509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.577574 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.577744 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.577744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.577951 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.578048 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.578050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.578834 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.578835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.579650 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.579652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.580141 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.580142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.580207 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.580735 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.580740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.581069 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.582769 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.582770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.582771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.583022 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.583216 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.583217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.583217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.583289 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.583523 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.583524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.583577 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.583665 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.583774 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.601689 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.594616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.0.98.62:35259)
0925 12:11:40.594676 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.594679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.594688 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:11:40.594806 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.594807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.594822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.594886 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.595491 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.596002 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.596768 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.596952 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.597047 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.599300 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.599301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.599302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.599309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.600662 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.600991 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.600992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.600993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.601046 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.601209 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.601209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.601209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.601331 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.601446 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.601561 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.601623 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:11:40.603330 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:40.604990 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:40.611652 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38679 (local address 127.0.0.1:48972)
0925 12:11:40.607511 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.607554 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.607623 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.607929 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.607933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.607936 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.608000 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.608003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609326 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.609327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.610377 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.610379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.611287 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.611288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.611405 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.611424 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.611501 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.611569 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":93,"thread_start_us":59,"threads_started":1}
I20260925 12:11:40.611670 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.0.98.2:38679)
0925 12:11:40.607671 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.607676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.607687 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:40.607729 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.607731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.607749 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.607837 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.608083 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609256 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.610469 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.611204 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.611285 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.611346 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.611528 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.611566 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260925 12:11:40.611917 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.0.1:46578)
0925 12:11:40.607959 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.607969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.608048 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.608302 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.608302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.608304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.608340 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.608341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609235 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.609236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.610565 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.610568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.611400 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.611401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.611447 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.611452 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.611501 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.611570 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":56,"thread_start_us":70,"threads_started":1}
I20260925 12:11:40.612630 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37947 (local address 127.0.0.1:54160)
0925 12:11:40.607730 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.607740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.607778 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.608021 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.608022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.608024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.608066 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.608068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609419 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.609419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.610579 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.610581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.612374 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.612375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.612448 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.612453 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.612521 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.612560 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260925 12:11:40.612632 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.0.98.1:34327)
0925 12:11:40.607897 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.607900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.608020 (+ 120us) server_negotiation.cc:407] Connection header received
0925 12:11:40.608142 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.608144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.608162 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.608217 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.608410 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609148 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.610654 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.611327 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.611403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.611445 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.611617 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.611656 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260925 12:11:40.612872 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.607580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.0.98.3:37947)
0925 12:11:40.607758 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.607762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.607772 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:11:40.607842 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.607844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.607857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.607959 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.608142 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.609256 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.611499 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.612191 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.612260 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.612341 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.612739 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.612780 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260925 12:11:40.612766 2399 tablet_service.cc:1467] Processing CreateTablet for tablet e272d873834b4bf7a29d284b07cc5af3 (DEFAULT_TABLE table=test-table [id=e2133347d51740eea108c6154db4c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.613155 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e272d873834b4bf7a29d284b07cc5af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.616382 2628 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Bootstrap starting.
I20260925 12:11:40.616848 2259 tablet_service.cc:1467] Processing CreateTablet for tablet e272d873834b4bf7a29d284b07cc5af3 (DEFAULT_TABLE table=test-table [id=e2133347d51740eea108c6154db4c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.617003 2628 tablet_bootstrap.cc:654] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.617180 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e272d873834b4bf7a29d284b07cc5af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.617340 2628 log.cc:824] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:40.618040 2628 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: No bootstrap required, opened a new log
I20260925 12:11:40.618095 2628 ts_tablet_manager.cc:1397] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:40.619669 2628 raft_consensus.cc:348] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.619800 2628 raft_consensus.cc:374] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.619822 2628 raft_consensus.cc:729] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.619948 2628 consensus_queue.cc:260] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.620194 2630 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Bootstrap starting.
I20260925 12:11:40.620249 2628 ts_tablet_manager.cc:1428] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:40.620358 2465 heartbeater.cc:503] Master 127.0.98.62:35259 was elected leader, sending a full tablet report...
I20260925 12:11:40.620865 2630 tablet_bootstrap.cc:654] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.620771 2531 tablet_service.cc:1467] Processing CreateTablet for tablet e272d873834b4bf7a29d284b07cc5af3 (DEFAULT_TABLE table=test-table [id=e2133347d51740eea108c6154db4c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.621100 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e272d873834b4bf7a29d284b07cc5af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.621191 2630 log.cc:824] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:40.621876 2630 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: No bootstrap required, opened a new log
I20260925 12:11:40.621958 2630 ts_tablet_manager.cc:1397] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:11:40.623235 2630 raft_consensus.cc:348] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.623379 2630 raft_consensus.cc:374] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.623435 2630 raft_consensus.cc:729] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.623545 2630 consensus_queue.cc:260] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.623838 2630 ts_tablet_manager.cc:1428] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:11:40.623873 2325 heartbeater.cc:503] Master 127.0.98.62:35259 was elected leader, sending a full tablet report...
I20260925 12:11:40.624055 2634 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Bootstrap starting.
I20260925 12:11:40.625018 2634 tablet_bootstrap.cc:654] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.625309 2634 log.cc:824] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:40.625977 2634 tablet_bootstrap.cc:492] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: No bootstrap required, opened a new log
I20260925 12:11:40.626060 2634 ts_tablet_manager.cc:1397] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:40.627350 2634 raft_consensus.cc:348] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.627497 2634 raft_consensus.cc:374] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.627542 2634 raft_consensus.cc:729] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac147094ddc4659a338cbb291f21775, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.627652 2634 consensus_queue.cc:260] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.627910 2634 ts_tablet_manager.cc:1428] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:40.627985 2605 heartbeater.cc:503] Master 127.0.98.62:35259 was elected leader, sending a full tablet report...
W20260925 12:11:40.662427 2466 tablet.cc:2369] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:11:40.742091 2326 tablet.cc:2369] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:40.811408 2636 raft_consensus.cc:482] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:40.811535 2636 raft_consensus.cc:504] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.811873 2636 leader_election.cc:288] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679)
I20260925 12:11:40.815770 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.812125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47359 (local address 127.0.98.1:34327)
0925 12:11:40.812172 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.812174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.812267 (+ 93us) server_negotiation.cc:407] Connection header received
0925 12:11:40.812342 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.812344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.812358 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.812402 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.812591 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.813389 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.814828 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.815392 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.815473 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.815484 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.815657 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.815701 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:40.815779 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.812104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.98.3:47359)
0925 12:11:40.812178 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.812190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.812274 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.812416 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.812417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.812418 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.812510 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.812511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.813466 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.813467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.814740 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.814742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.815488 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.815489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.815572 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.815578 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.815637 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.815685 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":4}
I20260925 12:11:40.815805 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.812195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47389 (local address 127.0.98.2:38679)
0925 12:11:40.812240 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.812242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.812494 (+ 252us) server_negotiation.cc:407] Connection header received
0925 12:11:40.812634 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.812635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.812647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.812719 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.812874 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.813555 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.814894 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.815416 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.815477 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.815485 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.815687 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.815722 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:11:40.815788 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.812106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38679 (local address 127.0.98.3:47389)
0925 12:11:40.812410 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.812423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.812539 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.812745 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.812746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.812748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.812802 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.812803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.813649 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.813649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.814823 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.814825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.815488 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.815489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.815572 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.815579 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.815679 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.815705 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":118,"thread_start_us":125,"threads_started":1}
I20260925 12:11:40.816187 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e272d873834b4bf7a29d284b07cc5af3" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23" is_pre_election: true
I20260925 12:11:40.816188 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e272d873834b4bf7a29d284b07cc5af3" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" is_pre_election: true
I20260925 12:11:40.816350 2279 raft_consensus.cc:2454] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 0.
I20260925 12:11:40.816350 2419 raft_consensus.cc:2454] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 0.
I20260925 12:11:40.816608 2490 leader_election.cc:302] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a64fffdd0a44d0aff4f5e910283b9c, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:40.816746 2636 raft_consensus.cc:2781] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:40.816813 2636 raft_consensus.cc:482] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:40.816838 2636 raft_consensus.cc:3037] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.817523 2636 raft_consensus.cc:504] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.817662 2636 leader_election.cc:288] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 election: Requested vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679)
I20260925 12:11:40.817829 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e272d873834b4bf7a29d284b07cc5af3" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23"
I20260925 12:11:40.817857 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e272d873834b4bf7a29d284b07cc5af3" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
I20260925 12:11:40.817914 2279 raft_consensus.cc:3037] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.817924 2419 raft_consensus.cc:3037] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.818759 2419 raft_consensus.cc:2454] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 1.
I20260925 12:11:40.818759 2279 raft_consensus.cc:2454] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 1.
I20260925 12:11:40.818920 2490 leader_election.cc:302] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a64fffdd0a44d0aff4f5e910283b9c, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:40.819002 2636 raft_consensus.cc:2781] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:40.819149 2636 raft_consensus.cc:686] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 1 LEADER]: Becoming Leader. State: Replica: aac147094ddc4659a338cbb291f21775, State: Running, Role: LEADER
I20260925 12:11:40.819245 2636 consensus_queue.cc:237] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } }
I20260925 12:11:40.820106 2121 catalog_manager.cc:5495] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 reported cstate change: term changed from 0 to 1, leader changed from <none> to aac147094ddc4659a338cbb291f21775 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aac147094ddc4659a338cbb291f21775" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } health_report { overall_health: UNKNOWN } } }
W20260925 12:11:40.823413 2606 tablet.cc:2369] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 12:11:40 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:40.855753 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.849165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35259 (local address 127.0.0.1:48210)
0925 12:11:40.849382 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.849394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.849491 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.849635 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.849636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.849962 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:40.850008 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.850010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.850660 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.850661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.851189 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.851190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.851530 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.851531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.851585 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.852083 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:40.852087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:40.853465 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:40.854850 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.854857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.854859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.855054 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.855226 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:40.855226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:40.855226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:40.855280 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:40.855490 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:40.855496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:40.855544 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.855599 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.855680 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260925 12:11:40.855898 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.849239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.98.62:35259)
0925 12:11:40.849368 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.849370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.849471 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:11:40.849544 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.849544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.849556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.849630 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:40.850095 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.850577 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.851273 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.851457 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.851533 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.853562 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:40.853563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:40.853563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:40.853571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:40.854783 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.855119 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.855119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.855120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.855167 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:40.855335 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:40.855336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:40.855336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:40.855436 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:40.855486 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.855663 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.855748 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:11:40.857788 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48210:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:11:40.858008 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:40.859488 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 12487f28a1c84466a322828f68f2eb35 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51f8df7a77904acbbe05f99f475caf85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:40.859560 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 12487f28a1c84466a322828f68f2eb35 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51f8df7a77904acbbe05f99f475caf85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:40.859663 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12487f28a1c84466a322828f68f2eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.859730 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12487f28a1c84466a322828f68f2eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.860659 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 12487f28a1c84466a322828f68f2eb35 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=51f8df7a77904acbbe05f99f475caf85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:11:40.860814 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12487f28a1c84466a322828f68f2eb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.861013 2630 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: Bootstrap starting.
I20260925 12:11:40.861034 2634 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: Bootstrap starting.
I20260925 12:11:40.861370 2630 tablet_bootstrap.cc:654] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.861418 2634 tablet_bootstrap.cc:654] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.861943 2630 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: No bootstrap required, opened a new log
I20260925 12:11:40.861975 2630 ts_tablet_manager.cc:1397] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:40.862092 2630 raft_consensus.cc:348] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.862139 2630 raft_consensus.cc:374] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.862151 2630 raft_consensus.cc:729] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.862211 2630 consensus_queue.cc:260] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.862270 2628 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: Bootstrap starting.
I20260925 12:11:40.862346 2630 ts_tablet_manager.cc:1428] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.862963 2628 tablet_bootstrap.cc:654] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.863061 2634 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: No bootstrap required, opened a new log
I20260925 12:11:40.863116 2634 ts_tablet_manager.cc:1397] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:40.863272 2634 raft_consensus.cc:348] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.863353 2634 raft_consensus.cc:374] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.863384 2634 raft_consensus.cc:729] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac147094ddc4659a338cbb291f21775, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.863430 2634 consensus_queue.cc:260] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.863407 2633 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.863480 2633 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.863528 2634 ts_tablet_manager.cc:1428] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:40.863583 2636 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.863701 2636 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.863773 2628 tablet_bootstrap.cc:492] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: No bootstrap required, opened a new log
I20260925 12:11:40.863819 2628 ts_tablet_manager.cc:1397] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:40.863972 2628 raft_consensus.cc:348] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.864039 2628 raft_consensus.cc:374] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:40.864073 2628 raft_consensus.cc:729] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Initialized, Role: FOLLOWER
I20260925 12:11:40.864122 2628 consensus_queue.cc:260] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.864212 2628 ts_tablet_manager.cc:1428] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:40.864346 2631 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.864460 2631 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.934684 2633 raft_consensus.cc:482] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:40.934777 2633 raft_consensus.cc:504] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.935106 2633 leader_election.cc:288] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:40.938299 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.935413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51577 (local address 127.0.98.2:38679)
0925 12:11:40.935473 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.935475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.935486 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:40.935517 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.935518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.935528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.935595 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.935772 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.936269 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.937343 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.937883 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.937937 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.937946 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.938174 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.938224 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:11:40.938345 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.935258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38679 (local address 127.0.98.1:51577)
0925 12:11:40.935396 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.935409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.935465 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.935605 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.935606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.935609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.935692 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.935694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.936355 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.936356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.937251 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.937254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.937989 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.937991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.938061 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.938065 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.938130 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.938172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260925 12:11:40.938716 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.935365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52249 (local address 127.0.98.3:37947)
0925 12:11:40.935415 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.935417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.935535 (+ 118us) server_negotiation.cc:407] Connection header received
0925 12:11:40.935598 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.935599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.935611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.935661 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:40.935839 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.936319 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.937812 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.938335 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.938406 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.938417 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.938613 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.938651 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:11:40.938766 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.935285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37947 (local address 127.0.98.1:52249)
0925 12:11:40.935444 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.935455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.935520 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.935678 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.935678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.935680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:40.935770 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.935771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.936410 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.936411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.937710 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.937713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.938453 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.938453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.938526 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.938529 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.938608 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.938644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":95,"thread_start_us":62,"threads_started":1}
I20260925 12:11:40.938731 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12487f28a1c84466a322828f68f2eb35" candidate_uuid: "30f220ccbab145efb3196809e86e1f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" is_pre_election: true
I20260925 12:11:40.938850 2419 raft_consensus.cc:2454] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f220ccbab145efb3196809e86e1f23 in term 0.
I20260925 12:11:40.939044 2210 leader_election.cc:302] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f220ccbab145efb3196809e86e1f23, 54a64fffdd0a44d0aff4f5e910283b9c; no voters:
I20260925 12:11:40.939198 2633 raft_consensus.cc:2781] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:40.939255 2633 raft_consensus.cc:482] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:40.939209 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12487f28a1c84466a322828f68f2eb35" candidate_uuid: "30f220ccbab145efb3196809e86e1f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775" is_pre_election: true
I20260925 12:11:40.939284 2633 raft_consensus.cc:3037] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.939333 2559 raft_consensus.cc:2454] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f220ccbab145efb3196809e86e1f23 in term 0.
I20260925 12:11:40.940032 2633 raft_consensus.cc:504] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.940171 2633 leader_election.cc:288] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [CANDIDATE]: Term 1 election: Requested vote from peers 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:40.940322 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12487f28a1c84466a322828f68f2eb35" candidate_uuid: "30f220ccbab145efb3196809e86e1f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775"
I20260925 12:11:40.940414 2559 raft_consensus.cc:3037] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.940423 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12487f28a1c84466a322828f68f2eb35" candidate_uuid: "30f220ccbab145efb3196809e86e1f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
I20260925 12:11:40.940569 2419 raft_consensus.cc:3037] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:40.941183 2559 raft_consensus.cc:2454] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f220ccbab145efb3196809e86e1f23 in term 1.
I20260925 12:11:40.941182 2419 raft_consensus.cc:2454] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f220ccbab145efb3196809e86e1f23 in term 1.
I20260925 12:11:40.941365 2210 leader_election.cc:302] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f220ccbab145efb3196809e86e1f23, 54a64fffdd0a44d0aff4f5e910283b9c; no voters:
I20260925 12:11:40.941468 2633 raft_consensus.cc:2781] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:40.941597 2633 raft_consensus.cc:686] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [term 1 LEADER]: Becoming Leader. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Running, Role: LEADER
I20260925 12:11:40.941704 2633 consensus_queue.cc:237] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:40.942168 2651 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: TxnStatusTablet state changed. Reason: New leader 30f220ccbab145efb3196809e86e1f23. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.942282 2651 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:40.942482 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:11:40.942618 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:11:40.942566 2121 catalog_manager.cc:5495] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30f220ccbab145efb3196809e86e1f23 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } health_report { overall_health: UNKNOWN } } }
I20260925 12:11:40.956806 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.953954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.0.1:46594)
0925 12:11:40.954020 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.954033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.954082 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.954321 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.954322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.954323 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:40.954376 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.954378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.955118 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.955119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.955787 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.955790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.956182 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.956183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.956260 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.956275 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.956620 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.956630 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.956676 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.956726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 12:11:40.956806 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.954013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.1:34327)
0925 12:11:40.954062 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.954065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.954092 (+ 27us) server_negotiation.cc:407] Connection header received
0925 12:11:40.954170 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.954171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.954190 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.954256 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:40.954478 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.955030 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.955887 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.956083 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.956195 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.956382 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.956543 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.956648 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.956694 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.956724 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:11:40.958592 2419 raft_consensus.cc:1261] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Refusing update from remote peer 30f220ccbab145efb3196809e86e1f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:40.958925 2559 raft_consensus.cc:1261] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Refusing update from remote peer 30f220ccbab145efb3196809e86e1f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:40.958942 2651 consensus_queue.cc:1035] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:40.959317 2651 consensus_queue.cc:1035] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:40.960147 2631 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: TxnStatusTablet state changed. Reason: New leader 30f220ccbab145efb3196809e86e1f23. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.960255 2631 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.960376 2636 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: TxnStatusTablet state changed. Reason: New leader 30f220ccbab145efb3196809e86e1f23. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.960444 2636 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.961222 2633 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.961304 2633 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:40.961372 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333225680723988480 to 7333225680657911808. Current Snapshot: MvccSnapshot[applied={T|T < 7333225680723988480}]
I20260925 12:11:40.961465 2636 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.961536 2636 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P aac147094ddc4659a338cbb291f21775: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.961784 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333225680723988480 to 7333225680657911808. Current Snapshot: MvccSnapshot[applied={T|T < 7333225680723988480}]
I20260925 12:11:40.962724 2651 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.962801 2651 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 30f220ccbab145efb3196809e86e1f23: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:11:40.963315 2631 tablet_replica.cc:439] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30f220ccbab145efb3196809e86e1f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } } }
I20260925 12:11:40.963398 2631 tablet_replica.cc:442] T 12487f28a1c84466a322828f68f2eb35 P 54a64fffdd0a44d0aff4f5e910283b9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:11:40.963534 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333225680723988480 to 7333225680657911808. Current Snapshot: MvccSnapshot[applied={T|T < 7333225680723988480 or (T in {7333225680723988480})}]
I20260925 12:11:40.966728 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.963951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.0.98.3:37947)
0925 12:11:40.963996 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.963998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.964125 (+ 127us) server_negotiation.cc:407] Connection header received
0925 12:11:40.964155 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.964155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.964172 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.964227 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:40.964403 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.964936 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.966039 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.966216 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.966265 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.966432 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.966519 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.966575 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.966637 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.966655 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:11:40.972090 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.969713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38379 (local address 127.0.98.1:34327)
0925 12:11:40.969752 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.969754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.969762 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:40.969858 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.969859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.969869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.969914 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:40.970063 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.970612 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.971360 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.971556 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.971636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.971842 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.971873 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.971940 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.972000 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.972016 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:40.972095 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.969511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.98.3:38379)
0925 12:11:40.969700 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.969713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.969761 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.969925 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.969926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.969927 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:40.969984 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.969986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.970693 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.970694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.971260 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.971262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.971638 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.971639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.971708 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.971721 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.971940 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.971942 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.971996 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.972033 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 12:11:40.974427 2419 raft_consensus.cc:1261] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Refusing update from remote peer aac147094ddc4659a338cbb291f21775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:40.974509 2279 raft_consensus.cc:1261] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Refusing update from remote peer aac147094ddc4659a338cbb291f21775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:40.974720 2636 consensus_queue.cc:1035] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:40.974834 2638 consensus_queue.cc:1035] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:40.983165 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.980621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34751 (local address 127.0.98.3:37947)
0925 12:11:40.980658 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:40.980660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:40.980794 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:11:40.980876 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:40.980877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:40.980888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:40.980931 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:40.981075 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.981536 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.982401 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.982698 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.982755 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.982933 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.982964 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.983017 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:40.983076 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:11:40.983093 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:11:40.983203 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:40.980583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37947 (local address 127.0.98.1:34751)
0925 12:11:40.980725 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:40.980737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:40.980821 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:40.980942 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:40.980943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:40.980945 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:40.981002 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.981003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.981615 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.981616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.982314 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:40.982317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:40.982753 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:40.982754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:40.982818 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:40.982834 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:40.983019 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:40.983021 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:11:40.983072 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:11:40.983111 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260925 12:11:40.992612 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:40.992736 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:40.993572 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Sending DeleteTablet for 3 replicas of tablet e272d873834b4bf7a29d284b07cc5af3
I20260925 12:11:40.994007 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet e272d873834b4bf7a29d284b07cc5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48972
I20260925 12:11:40.994261 2675 tablet_replica.cc:330] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: stopping tablet replica
I20260925 12:11:40.994352 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet e272d873834b4bf7a29d284b07cc5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 12:11:40.994436 2675 raft_consensus.cc:2229] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:40.994498 2675 raft_consensus.cc:2258] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:40.994649 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet e272d873834b4bf7a29d284b07cc5af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46578
I20260925 12:11:40.994925 2675 ts_tablet_manager.cc:1905] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:40.994925 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:40.995064 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:40.996709 2399 tablet_service.cc:1467] Processing CreateTablet for tablet b49d7e498e3e441197dbcec07674be50 (DEFAULT_TABLE table=test-table [id=7ad1143b84834600aee97f6a1d641de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.996797 2531 tablet_service.cc:1467] Processing CreateTablet for tablet b49d7e498e3e441197dbcec07674be50 (DEFAULT_TABLE table=test-table [id=7ad1143b84834600aee97f6a1d641de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.996872 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49d7e498e3e441197dbcec07674be50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.996929 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49d7e498e3e441197dbcec07674be50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:40.998013 2628 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Bootstrap starting.
I20260925 12:11:40.998227 2634 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Bootstrap starting.
I20260925 12:11:40.998389 2628 tablet_bootstrap.cc:654] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.998610 2634 tablet_bootstrap.cc:654] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:40.998690 2675 ts_tablet_manager.cc:1918] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:40.998747 2675 log.cc:1196] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/e272d873834b4bf7a29d284b07cc5af3
I20260925 12:11:40.999013 2675 ts_tablet_manager.cc:1939] T e272d873834b4bf7a29d284b07cc5af3 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting consensus metadata
I20260925 12:11:40.999408 2106 catalog_manager.cc:4841] TS 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679): tablet e272d873834b4bf7a29d284b07cc5af3 (table test-table [id=e2133347d51740eea108c6154db4c1a8]) successfully deleted
I20260925 12:11:40.999488 2259 tablet_service.cc:1467] Processing CreateTablet for tablet b49d7e498e3e441197dbcec07674be50 (DEFAULT_TABLE table=test-table [id=7ad1143b84834600aee97f6a1d641de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:40.999607 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49d7e498e3e441197dbcec07674be50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:41.000619 2677 tablet_replica.cc:330] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: stopping tablet replica
I20260925 12:11:41.000778 2677 raft_consensus.cc:2229] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:41.000828 2677 raft_consensus.cc:2258] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:41.001032 2630 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Bootstrap starting.
I20260925 12:11:41.001399 2630 tablet_bootstrap.cc:654] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:41.001801 2634 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: No bootstrap required, opened a new log
I20260925 12:11:41.001869 2634 ts_tablet_manager.cc:1397] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 12:11:41.002043 2634 raft_consensus.cc:348] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.002110 2634 raft_consensus.cc:374] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:41.002130 2634 raft_consensus.cc:729] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac147094ddc4659a338cbb291f21775, State: Initialized, Role: FOLLOWER
I20260925 12:11:41.002179 2634 consensus_queue.cc:260] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.002271 2634 ts_tablet_manager.cc:1428] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:41.002621 2676 tablet_replica.cc:330] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: stopping tablet replica
I20260925 12:11:41.003095 2676 raft_consensus.cc:2229] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:41.003217 2676 raft_consensus.cc:2258] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:41.003873 2676 ts_tablet_manager.cc:1905] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:41.005510 2676 ts_tablet_manager.cc:1918] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:41.005631 2676 log.cc:1196] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/e272d873834b4bf7a29d284b07cc5af3
I20260925 12:11:41.005946 2676 ts_tablet_manager.cc:1939] T e272d873834b4bf7a29d284b07cc5af3 P aac147094ddc4659a338cbb291f21775: Deleting consensus metadata
I20260925 12:11:41.006186 2628 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: No bootstrap required, opened a new log
I20260925 12:11:41.006243 2628 ts_tablet_manager.cc:1397] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 12:11:41.006389 2628 raft_consensus.cc:348] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.006458 2628 raft_consensus.cc:374] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:41.006479 2628 raft_consensus.cc:729] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Initialized, Role: FOLLOWER
I20260925 12:11:41.006521 2628 consensus_queue.cc:260] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.006598 2628 ts_tablet_manager.cc:1428] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:41.007608 2109 catalog_manager.cc:4841] TS aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947): tablet e272d873834b4bf7a29d284b07cc5af3 (table test-table [id=e2133347d51740eea108c6154db4c1a8]) successfully deleted
I20260925 12:11:41.007890 2630 tablet_bootstrap.cc:492] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: No bootstrap required, opened a new log
I20260925 12:11:41.007943 2630 ts_tablet_manager.cc:1397] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 12:11:41.008088 2630 raft_consensus.cc:348] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.008160 2630 raft_consensus.cc:374] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:41.008179 2630 raft_consensus.cc:729] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Initialized, Role: FOLLOWER
I20260925 12:11:41.008227 2630 consensus_queue.cc:260] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.008317 2630 ts_tablet_manager.cc:1428] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:41.012758 2677 ts_tablet_manager.cc:1905] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:41.014326 2677 ts_tablet_manager.cc:1918] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:41.014426 2677 log.cc:1196] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/e272d873834b4bf7a29d284b07cc5af3
I20260925 12:11:41.014717 2677 ts_tablet_manager.cc:1939] T e272d873834b4bf7a29d284b07cc5af3 P 30f220ccbab145efb3196809e86e1f23: Deleting consensus metadata
I20260925 12:11:41.015170 2107 catalog_manager.cc:4841] TS 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327): tablet e272d873834b4bf7a29d284b07cc5af3 (table test-table [id=e2133347d51740eea108c6154db4c1a8]) successfully deleted
I20260925 12:11:41.275334 2669 raft_consensus.cc:482] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:41.275463 2669 raft_consensus.cc:504] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.275673 2669 leader_election.cc:288] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679)
I20260925 12:11:41.275913 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49d7e498e3e441197dbcec07674be50" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23" is_pre_election: true
I20260925 12:11:41.275913 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49d7e498e3e441197dbcec07674be50" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" is_pre_election: true
I20260925 12:11:41.276026 2419 raft_consensus.cc:2454] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 0.
I20260925 12:11:41.276026 2279 raft_consensus.cc:2454] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 0.
I20260925 12:11:41.276190 2491 leader_election.cc:302] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f220ccbab145efb3196809e86e1f23, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:41.276343 2669 raft_consensus.cc:2781] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:41.276400 2669 raft_consensus.cc:482] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:41.276437 2669 raft_consensus.cc:3037] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:41.277253 2669 raft_consensus.cc:504] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.277400 2669 leader_election.cc:288] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 election: Requested vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679)
I20260925 12:11:41.277573 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49d7e498e3e441197dbcec07674be50" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23"
I20260925 12:11:41.277606 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49d7e498e3e441197dbcec07674be50" candidate_uuid: "aac147094ddc4659a338cbb291f21775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a64fffdd0a44d0aff4f5e910283b9c"
I20260925 12:11:41.277676 2279 raft_consensus.cc:3037] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:41.277675 2419 raft_consensus.cc:3037] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:41.278261 2419 raft_consensus.cc:2454] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 1.
I20260925 12:11:41.278302 2279 raft_consensus.cc:2454] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac147094ddc4659a338cbb291f21775 in term 1.
I20260925 12:11:41.278463 2490 leader_election.cc:302] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a64fffdd0a44d0aff4f5e910283b9c, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:41.278582 2669 raft_consensus.cc:2781] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:41.278628 2669 raft_consensus.cc:686] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 1 LEADER]: Becoming Leader. State: Replica: aac147094ddc4659a338cbb291f21775, State: Running, Role: LEADER
I20260925 12:11:41.278681 2669 consensus_queue.cc:237] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:41.279291 2122 catalog_manager.cc:5495] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 reported cstate change: term changed from 0 to 1, leader changed from <none> to aac147094ddc4659a338cbb291f21775 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aac147094ddc4659a338cbb291f21775" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } health_report { overall_health: HEALTHY } } }
I20260925 12:11:41.290508 2279 raft_consensus.cc:1261] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Refusing update from remote peer aac147094ddc4659a338cbb291f21775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:41.290529 2419 raft_consensus.cc:1261] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Refusing update from remote peer aac147094ddc4659a338cbb291f21775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:41.290728 2669 consensus_queue.cc:1035] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:41.290832 2636 consensus_queue.cc:1035] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:11:41.295009 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 12:11:42.812161 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:42.812273 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:42.813145 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Sending DeleteTablet for 3 replicas of tablet b49d7e498e3e441197dbcec07674be50
I20260925 12:11:42.813381 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet b49d7e498e3e441197dbcec07674be50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46578
I20260925 12:11:42.813418 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet b49d7e498e3e441197dbcec07674be50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 12:11:42.813638 2706 tablet_replica.cc:330] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: stopping tablet replica
I20260925 12:11:42.813701 2706 raft_consensus.cc:2229] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:42.813750 2706 raft_consensus.cc:2258] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.813958 2706 ts_tablet_manager.cc:1905] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:42.814922 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet b49d7e498e3e441197dbcec07674be50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48972
I20260925 12:11:42.815213 2708 tablet_replica.cc:330] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: stopping tablet replica
I20260925 12:11:42.815281 2708 raft_consensus.cc:2229] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:42.815322 2708 raft_consensus.cc:2258] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.815534 2708 ts_tablet_manager.cc:1905] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:42.818512 2705 tablet_replica.cc:330] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: stopping tablet replica
I20260925 12:11:42.818604 2705 raft_consensus.cc:2229] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:42.818679 2705 raft_consensus.cc:2258] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.818920 2705 ts_tablet_manager.cc:1905] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 12:11:42 dist-test-slave-2hqq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 12:11:42.849808 2705 ts_tablet_manager.cc:1918] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:11:42.849808 2708 ts_tablet_manager.cc:1918] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:11:42.849897 2705 log.cc:1196] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/b49d7e498e3e441197dbcec07674be50
I20260925 12:11:42.849947 2706 ts_tablet_manager.cc:1918] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:11:42.850023 2706 log.cc:1196] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/b49d7e498e3e441197dbcec07674be50
I20260925 12:11:42.850142 2708 log.cc:1196] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/b49d7e498e3e441197dbcec07674be50
I20260925 12:11:42.850203 2705 ts_tablet_manager.cc:1939] T b49d7e498e3e441197dbcec07674be50 P aac147094ddc4659a338cbb291f21775: Deleting consensus metadata
I20260925 12:11:42.850260 2706 ts_tablet_manager.cc:1939] T b49d7e498e3e441197dbcec07674be50 P 30f220ccbab145efb3196809e86e1f23: Deleting consensus metadata
I20260925 12:11:42.850497 2708 ts_tablet_manager.cc:1939] T b49d7e498e3e441197dbcec07674be50 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting consensus metadata
I20260925 12:11:42.850917 2107 catalog_manager.cc:4841] TS 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327): tablet b49d7e498e3e441197dbcec07674be50 (table test-table [id=7ad1143b84834600aee97f6a1d641de9]) successfully deleted
I20260925 12:11:42.850927 2109 catalog_manager.cc:4841] TS aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947): tablet b49d7e498e3e441197dbcec07674be50 (table test-table [id=7ad1143b84834600aee97f6a1d641de9]) successfully deleted
I20260925 12:11:42.851220 2106 catalog_manager.cc:4841] TS 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679): tablet b49d7e498e3e441197dbcec07674be50 (table test-table [id=7ad1143b84834600aee97f6a1d641de9]) successfully deleted
Sep 25 12:11:42 dist-test-slave-2hqq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:42.859798 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.850597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.62:35259)
0925 12:11:42.850950 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.850952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.851153 (+ 201us) server_negotiation.cc:407] Connection header received
0925 12:11:42.851230 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.851231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.851245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.851337 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:42.852216 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.852815 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.854109 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.854309 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.854378 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.857217 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:42.857220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:42.857220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:42.857229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:42.858733 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:42.859104 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:42.859105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:42.859106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:42.859159 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:42.859329 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:42.859330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:42.859330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:42.859446 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:42.859527 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.859662 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.859728 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":82,"threads_started":1}
I20260925 12:11:42.861600 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:42.861774 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:42.863333 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 84807beb415247b2adc0c6e7c1689867 (DEFAULT_TABLE table=test-table [id=be37850f44d040c29f37db114432a460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:42.863344 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 84807beb415247b2adc0c6e7c1689867 (DEFAULT_TABLE table=test-table [id=be37850f44d040c29f37db114432a460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:42.863338 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 84807beb415247b2adc0c6e7c1689867 (DEFAULT_TABLE table=test-table [id=be37850f44d040c29f37db114432a460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:42.863482 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84807beb415247b2adc0c6e7c1689867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:42.863481 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84807beb415247b2adc0c6e7c1689867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:42.863492 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84807beb415247b2adc0c6e7c1689867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:42.864997 2718 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Bootstrap starting.
I20260925 12:11:42.865096 2716 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Bootstrap starting.
I20260925 12:11:42.865135 2717 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Bootstrap starting.
I20260925 12:11:42.865448 2716 tablet_bootstrap.cc:654] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:42.865475 2717 tablet_bootstrap.cc:654] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:42.866083 2716 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: No bootstrap required, opened a new log
I20260925 12:11:42.866079 2718 tablet_bootstrap.cc:654] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:42.866139 2716 ts_tablet_manager.cc:1397] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:42.866295 2716 raft_consensus.cc:348] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.866382 2716 raft_consensus.cc:374] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:42.866402 2716 raft_consensus.cc:729] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Initialized, Role: FOLLOWER
I20260925 12:11:42.866480 2716 consensus_queue.cc:260] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.866644 2716 ts_tablet_manager.cc:1428] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:42.866880 2717 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: No bootstrap required, opened a new log
I20260925 12:11:42.866909 2718 tablet_bootstrap.cc:492] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: No bootstrap required, opened a new log
I20260925 12:11:42.866938 2717 ts_tablet_manager.cc:1397] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:42.866948 2718 ts_tablet_manager.cc:1397] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:42.867057 2718 raft_consensus.cc:348] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.867075 2717 raft_consensus.cc:348] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.867131 2718 raft_consensus.cc:374] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:42.867136 2717 raft_consensus.cc:374] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:42.867164 2718 raft_consensus.cc:729] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Initialized, Role: FOLLOWER
I20260925 12:11:42.867172 2717 raft_consensus.cc:729] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac147094ddc4659a338cbb291f21775, State: Initialized, Role: FOLLOWER
I20260925 12:11:42.867233 2717 consensus_queue.cc:260] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.867233 2718 consensus_queue.cc:260] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.867333 2717 ts_tablet_manager.cc:1428] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:42.867465 2718 ts_tablet_manager.cc:1428] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:42.867676 2722 raft_consensus.cc:482] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:42.867799 2722 raft_consensus.cc:504] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.868304 2722 leader_election.cc:288] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:42.871486 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.868587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48209 (local address 127.0.98.1:34327)
0925 12:11:42.868720 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.868721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.868762 (+ 41us) server_negotiation.cc:407] Connection header received
0925 12:11:42.868818 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.868819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.868827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.868908 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:42.869063 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.869561 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.870601 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.871179 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.871237 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.871247 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.871405 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.871435 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":91,"threads_started":1}
I20260925 12:11:42.871754 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.98.2:48209)
0925 12:11:42.868707 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:42.868716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:42.868765 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:42.868912 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:42.868914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:42.868916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:42.868982 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.868984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.869641 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.869642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.870510 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.870512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.871256 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.871257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.871332 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.871336 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:42.871394 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 12:11:42.871434 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":28,"thread_start_us":124,"threads_started":1}
I20260925 12:11:42.871992 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84807beb415247b2adc0c6e7c1689867" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23" is_pre_election: true
I20260925 12:11:42.872056 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.868596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46923 (local address 127.0.98.3:37947)
0925 12:11:42.868860 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.868862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.868869 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:11:42.868890 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.868891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.868900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.868961 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:42.869151 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.869637 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.871118 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.871655 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.871729 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.871742 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.871942 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.871981 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":74,"threads_started":1}
I20260925 12:11:42.872062 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.868428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37947 (local address 127.0.98.2:46923)
0925 12:11:42.868623 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:42.868636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:42.868701 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:42.869003 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:42.869004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:42.869007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:42.869079 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.869080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.869713 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.869713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.871033 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.871035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.871776 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.871777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.871850 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.871854 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:42.871937 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:11:42.871977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260925 12:11:42.872076 2279 raft_consensus.cc:2454] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 0.
I20260925 12:11:42.872287 2351 leader_election.cc:302] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f220ccbab145efb3196809e86e1f23, 54a64fffdd0a44d0aff4f5e910283b9c; no voters:
I20260925 12:11:42.872301 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84807beb415247b2adc0c6e7c1689867" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775" is_pre_election: true
I20260925 12:11:42.872391 2559 raft_consensus.cc:2454] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 0.
I20260925 12:11:42.872431 2722 raft_consensus.cc:2781] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:42.872467 2722 raft_consensus.cc:482] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:42.872519 2722 raft_consensus.cc:3037] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:42.938787 2722 raft_consensus.cc:504] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.938992 2722 leader_election.cc:288] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 election: Requested vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:42.939208 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84807beb415247b2adc0c6e7c1689867" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23"
I20260925 12:11:42.939230 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84807beb415247b2adc0c6e7c1689867" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775"
I20260925 12:11:42.939312 2279 raft_consensus.cc:3037] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:42.939337 2559 raft_consensus.cc:3037] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:42.939989 2279 raft_consensus.cc:2454] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 1.
I20260925 12:11:42.940209 2351 leader_election.cc:302] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f220ccbab145efb3196809e86e1f23, 54a64fffdd0a44d0aff4f5e910283b9c; no voters:
I20260925 12:11:42.940269 2559 raft_consensus.cc:2454] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 1.
I20260925 12:11:42.940332 2722 raft_consensus.cc:2781] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:42.940531 2722 raft_consensus.cc:686] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 LEADER]: Becoming Leader. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Running, Role: LEADER
I20260925 12:11:42.940618 2722 consensus_queue.cc:237] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:42.941416 2121 catalog_manager.cc:5495] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } health_report { overall_health: UNKNOWN } } }
I20260925 12:11:42.961956 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.959449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.0.98.2:38679)
0925 12:11:42.959594 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.959597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.959605 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:42.959657 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.959658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.959678 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.959749 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:42.959830 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.960373 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.961166 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.961419 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.961475 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.961610 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:42.961726 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:42.961782 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.961816 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.961832 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 12:11:42.966449 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.964334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34027 (local address 127.0.98.1:34327)
0925 12:11:42.964377 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.964379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.964477 (+ 98us) server_negotiation.cc:407] Connection header received
0925 12:11:42.964555 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.964555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.964566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.964626 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:42.964739 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.965183 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.965872 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.966052 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.966102 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.966261 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:42.966285 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:42.966323 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.966374 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.966392 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:42.966465 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.964250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.98.2:34027)
0925 12:11:42.964396 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:42.964407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:42.964468 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:42.964642 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:42.964642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:42.964644 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:42.964694 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.964695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.965236 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.965236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.965795 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.965796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.966114 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.966115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.966178 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.966191 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:42.966335 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:42.966336 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:42.966367 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 12:11:42.966400 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 12:11:42.968988 2559 raft_consensus.cc:1261] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Refusing update from remote peer 54a64fffdd0a44d0aff4f5e910283b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:42.968988 2279 raft_consensus.cc:1261] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Refusing update from remote peer 54a64fffdd0a44d0aff4f5e910283b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:42.969282 2722 consensus_queue.cc:1035] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:42.969421 2727 consensus_queue.cc:1035] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:42.977135 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.974782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44563 (local address 127.0.98.2:38679)
0925 12:11:42.974816 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:42.974818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:42.974828 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:11:42.974853 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:42.974854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:42.974864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:42.974913 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:42.975023 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.975533 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.976391 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.976672 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.976751 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.976889 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:42.976923 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:42.976964 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:42.977027 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:11:42.977046 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:11:42.977160 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:42.974619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38679 (local address 127.0.98.1:44563)
0925 12:11:42.974747 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:42.974758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:42.974806 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:42.974915 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:42.974916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:42.974917 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:11:42.974967 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.974969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.975604 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.975605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.976308 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:42.976311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:42.976761 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:42.976762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:42.976821 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:42.976826 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:11:42.976974 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:11:42.976975 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:42.977022 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 12:11:42.977065 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260925 12:11:42.986372 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:42.986482 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:42.987228 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Sending DeleteTablet for 3 replicas of tablet 84807beb415247b2adc0c6e7c1689867
I20260925 12:11:42.987496 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet 84807beb415247b2adc0c6e7c1689867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 12:11:42.987500 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 84807beb415247b2adc0c6e7c1689867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46578
I20260925 12:11:42.987593 2705 tablet_replica.cc:330] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: stopping tablet replica
I20260925 12:11:42.987593 2706 tablet_replica.cc:330] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: stopping tablet replica
I20260925 12:11:42.987641 2705 raft_consensus.cc:2229] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:42.987645 2706 raft_consensus.cc:2229] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:42.987677 2705 raft_consensus.cc:2258] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.987695 2706 raft_consensus.cc:2258] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.987980 2705 ts_tablet_manager.cc:1905] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:42.987979 2706 ts_tablet_manager.cc:1905] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:42.988981 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 84807beb415247b2adc0c6e7c1689867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48972
I20260925 12:11:42.989086 2708 tablet_replica.cc:330] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: stopping tablet replica
I20260925 12:11:42.989138 2708 raft_consensus.cc:2229] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:42.989231 2708 raft_consensus.cc:2258] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:42.989228 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:42.989415 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:42.989557 2708 ts_tablet_manager.cc:1905] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:42.990574 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 71a46ed28c844b6c82322e7bce57007e (DEFAULT_TABLE table=test-table [id=01b42f167d6d486b8581492eefc8ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:42.990604 2531 tablet_service.cc:1467] Processing CreateTablet for tablet 71a46ed28c844b6c82322e7bce57007e (DEFAULT_TABLE table=test-table [id=01b42f167d6d486b8581492eefc8ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:42.990708 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 71a46ed28c844b6c82322e7bce57007e (DEFAULT_TABLE table=test-table [id=01b42f167d6d486b8581492eefc8ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:43.039866 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a46ed28c844b6c82322e7bce57007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:43.039866 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a46ed28c844b6c82322e7bce57007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:43.041322 2716 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: Bootstrap starting.
I20260925 12:11:43.041544 2717 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Bootstrap starting.
I20260925 12:11:43.041639 2716 tablet_bootstrap.cc:654] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:43.041873 2717 tablet_bootstrap.cc:654] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:43.042078 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a46ed28c844b6c82322e7bce57007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:43.042871 2705 ts_tablet_manager.cc:1918] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:43.042925 2705 log.cc:1196] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/84807beb415247b2adc0c6e7c1689867
I20260925 12:11:43.043149 2705 ts_tablet_manager.cc:1939] T 84807beb415247b2adc0c6e7c1689867 P aac147094ddc4659a338cbb291f21775: Deleting consensus metadata
I20260925 12:11:43.043560 2718 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: Bootstrap starting.
I20260925 12:11:43.043605 2717 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: No bootstrap required, opened a new log
I20260925 12:11:43.043651 2717 ts_tablet_manager.cc:1397] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:43.043821 2717 raft_consensus.cc:348] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.043902 2717 raft_consensus.cc:374] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:43.043917 2718 tablet_bootstrap.cc:654] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:43.043927 2717 raft_consensus.cc:729] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac147094ddc4659a338cbb291f21775, State: Initialized, Role: FOLLOWER
I20260925 12:11:43.043979 2717 consensus_queue.cc:260] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.044238 2717 ts_tablet_manager.cc:1428] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:43.045558 2716 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: No bootstrap required, opened a new log
I20260925 12:11:43.045620 2716 ts_tablet_manager.cc:1397] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 12:11:43.045653 2718 tablet_bootstrap.cc:492] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: No bootstrap required, opened a new log
I20260925 12:11:43.045699 2718 ts_tablet_manager.cc:1397] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:11:43.045761 2716 raft_consensus.cc:348] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.045848 2716 raft_consensus.cc:374] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:43.045825 2718 raft_consensus.cc:348] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.045871 2716 raft_consensus.cc:729] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Initialized, Role: FOLLOWER
I20260925 12:11:43.045890 2718 raft_consensus.cc:374] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:43.045912 2718 raft_consensus.cc:729] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f220ccbab145efb3196809e86e1f23, State: Initialized, Role: FOLLOWER
I20260925 12:11:43.045914 2716 consensus_queue.cc:260] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.045955 2718 consensus_queue.cc:260] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.045994 2716 ts_tablet_manager.cc:1428] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:43.046061 2718 ts_tablet_manager.cc:1428] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:43.072651 2706 ts_tablet_manager.cc:1918] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:43.072757 2706 log.cc:1196] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/84807beb415247b2adc0c6e7c1689867
I20260925 12:11:43.073004 2706 ts_tablet_manager.cc:1939] T 84807beb415247b2adc0c6e7c1689867 P 30f220ccbab145efb3196809e86e1f23: Deleting consensus metadata
I20260925 12:11:43.097297 2109 catalog_manager.cc:4841] TS aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947): tablet 84807beb415247b2adc0c6e7c1689867 (table test-table [id=be37850f44d040c29f37db114432a460]) successfully deleted
I20260925 12:11:43.120873 2107 catalog_manager.cc:4841] TS 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327): tablet 84807beb415247b2adc0c6e7c1689867 (table test-table [id=be37850f44d040c29f37db114432a460]) successfully deleted
I20260925 12:11:43.135105 2708 ts_tablet_manager.cc:1918] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:11:43.135193 2708 log.cc:1196] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/84807beb415247b2adc0c6e7c1689867
I20260925 12:11:43.135437 2708 ts_tablet_manager.cc:1939] T 84807beb415247b2adc0c6e7c1689867 P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting consensus metadata
I20260925 12:11:43.135885 2106 catalog_manager.cc:4841] TS 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2:38679): tablet 84807beb415247b2adc0c6e7c1689867 (table test-table [id=be37850f44d040c29f37db114432a460]) successfully deleted
I20260925 12:11:43.203842 2727 raft_consensus.cc:482] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:43.203966 2727 raft_consensus.cc:504] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.204164 2727 leader_election.cc:288] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:43.204512 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a46ed28c844b6c82322e7bce57007e" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775" is_pre_election: true
I20260925 12:11:43.204512 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a46ed28c844b6c82322e7bce57007e" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23" is_pre_election: true
I20260925 12:11:43.204629 2279 raft_consensus.cc:2454] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 0.
I20260925 12:11:43.204629 2559 raft_consensus.cc:2454] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 0.
I20260925 12:11:43.204787 2353 leader_election.cc:302] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a64fffdd0a44d0aff4f5e910283b9c, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:43.204916 2727 raft_consensus.cc:2781] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:43.204954 2727 raft_consensus.cc:482] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:43.204977 2727 raft_consensus.cc:3037] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:43.205792 2727 raft_consensus.cc:504] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.205940 2727 leader_election.cc:288] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 election: Requested vote from peers 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327), aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947)
I20260925 12:11:43.206100 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a46ed28c844b6c82322e7bce57007e" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f220ccbab145efb3196809e86e1f23"
I20260925 12:11:43.206135 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a46ed28c844b6c82322e7bce57007e" candidate_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac147094ddc4659a338cbb291f21775"
I20260925 12:11:43.206218 2559 raft_consensus.cc:3037] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:43.206223 2279 raft_consensus.cc:3037] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:43.206813 2559 raft_consensus.cc:2454] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 1.
I20260925 12:11:43.206812 2279 raft_consensus.cc:2454] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54a64fffdd0a44d0aff4f5e910283b9c in term 1.
I20260925 12:11:43.206982 2353 leader_election.cc:302] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54a64fffdd0a44d0aff4f5e910283b9c, aac147094ddc4659a338cbb291f21775; no voters:
I20260925 12:11:43.207116 2727 raft_consensus.cc:2781] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:43.207194 2727 raft_consensus.cc:686] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 LEADER]: Becoming Leader. State: Replica: 54a64fffdd0a44d0aff4f5e910283b9c, State: Running, Role: LEADER
I20260925 12:11:43.207248 2727 consensus_queue.cc:237] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } }
I20260925 12:11:43.207865 2122 catalog_manager.cc:5495] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 54a64fffdd0a44d0aff4f5e910283b9c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54a64fffdd0a44d0aff4f5e910283b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 } health_report { overall_health: UNKNOWN } } }
I20260925 12:11:43.222584 2559 raft_consensus.cc:1261] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Refusing update from remote peer 54a64fffdd0a44d0aff4f5e910283b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:43.222599 2279 raft_consensus.cc:1261] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Refusing update from remote peer 54a64fffdd0a44d0aff4f5e910283b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:43.222832 2727 consensus_queue.cc:1035] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f220ccbab145efb3196809e86e1f23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:43.222930 2738 consensus_queue.cc:1035] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac147094ddc4659a338cbb291f21775" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:11:43.227486 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 12:11:44.745707 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:44.745826 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:44.746831 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef32fbe20c34c3a95b3202c72511516: Sending DeleteTablet for 3 replicas of tablet 71a46ed28c844b6c82322e7bce57007e
I20260925 12:11:44.747136 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a46ed28c844b6c82322e7bce57007e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 12:11:44.747164 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a46ed28c844b6c82322e7bce57007e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46578
I20260925 12:11:44.747356 2767 tablet_replica.cc:330] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: stopping tablet replica
I20260925 12:11:44.747392 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a46ed28c844b6c82322e7bce57007e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48972
I20260925 12:11:44.747409 2768 tablet_replica.cc:330] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23: stopping tablet replica
I20260925 12:11:44.747458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2194
I20260925 12:11:44.747491 2767 raft_consensus.cc:2229] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:44.747514 2768 raft_consensus.cc:2229] T 71a46ed28c844b6c82322e7bce57007e P 30f220ccbab145efb3196809e86e1f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:44.748384 2767 raft_consensus.cc:2258] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:44.748687 2767 ts_tablet_manager.cc:1905] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:44.750468 2767 ts_tablet_manager.cc:1918] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 12:11:44.750519 2767 log.cc:1196] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/71a46ed28c844b6c82322e7bce57007e
I20260925 12:11:44.750734 2767 ts_tablet_manager.cc:1939] T 71a46ed28c844b6c82322e7bce57007e P aac147094ddc4659a338cbb291f21775: Deleting consensus metadata
I20260925 12:11:44.751102 2109 catalog_manager.cc:4841] TS aac147094ddc4659a338cbb291f21775 (127.0.98.3:37947): tablet 71a46ed28c844b6c82322e7bce57007e (table test-table [id=01b42f167d6d486b8581492eefc8ae48]) successfully deleted
I20260925 12:11:44.752599 2769 tablet_replica.cc:330] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: stopping tablet replica
I20260925 12:11:44.752656 2769 raft_consensus.cc:2229] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:44.752728 2769 raft_consensus.cc:2258] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:44.752990 2769 ts_tablet_manager.cc:1905] T 71a46ed28c844b6c82322e7bce57007e P 54a64fffdd0a44d0aff4f5e910283b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 12:11:44.754087 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34327 (error 108)
I20260925 12:11:44.754308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2334
I20260925 12:11:44.756575 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:44.756290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34327 (local address 127.0.0.1:35404)
0925 12:11:44.756440 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:44.756534 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
W20260925 12:11:44.756731 2107 catalog_manager.cc:4584] TS 30f220ccbab145efb3196809e86e1f23 (127.0.98.1:34327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71a46ed28c844b6c82322e7bce57007e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34327: connect: Connection refused (error 111)
I20260925 12:11:44.760525 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:44.760374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38679 (local address 127.0.0.1:36204)
0925 12:11:44.760413 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:44.760436 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 12:11:44.760607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2474
I20260925 12:11:44.766281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2090
2026-09-25T12:11:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6046 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:44 dist-test-slave-2hqq krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:44 dist-test-slave-2hqq krb5kdc[2774](info): set up 2 sockets
Sep 25 12:11:44 dist-test-slave-2hqq krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260925 12:11:45.812330 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:45 dist-test-slave-2hqq krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:45.932132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:46.016849 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:46.017087 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:46.017122 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:46.018347 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:46.018407 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:46.018430 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:46.018457 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:46.018487 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:46.019974 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:46.020231 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:46.020453 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:11:46.025275 2787 server_base.cc:1034] running on GCE node
W20260925 12:11:46.025352 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.025362 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.025449 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:46.025745 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:46.026034 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:46.027174 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790338306027160 us; error 25 us; skew 500 ppm
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:46.029634 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:46.030074 2787 webserver.cc:460] Webserver started at http://127.0.98.62:37697/ using document root <none> and password file <none>
I20260925 12:11:46.030301 2787 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:46.030369 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:46.030517 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:46.031595 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "d96e9f45331d4ce9bba367ae4cdc7700"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "0483da82c754ab9d7255e3e190bbb260"
server_key_iv: "82986728dab9cd15bfa2ff3e8b2b0354"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.031955 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "d96e9f45331d4ce9bba367ae4cdc7700"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "0483da82c754ab9d7255e3e190bbb260"
server_key_iv: "82986728dab9cd15bfa2ff3e8b2b0354"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.033380 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:46.034076 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.034245 2787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.034341 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "d96e9f45331d4ce9bba367ae4cdc7700"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "0483da82c754ab9d7255e3e190bbb260"
server_key_iv: "82986728dab9cd15bfa2ff3e8b2b0354"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.034425 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:46.050987 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:46.053632 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:46.053823 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:46.057951 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37723
I20260925 12:11:46.057984 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37723 every 8 connection(s)
I20260925 12:11:46.058487 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:46.059309 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:46.061728 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700: Bootstrap starting.
I20260925 12:11:46.062422 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:46.062722 2855 log.cc:824] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:46.063481 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700: No bootstrap required, opened a new log
I20260925 12:11:46.065425 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } }
I20260925 12:11:46.065568 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:46.065591 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96e9f45331d4ce9bba367ae4cdc7700, State: Initialized, Role: FOLLOWER
I20260925 12:11:46.065744 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } }
I20260925 12:11:46.065828 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:46.065851 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:46.065919 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:46.066821 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } }
I20260925 12:11:46.066910 2855 leader_election.cc:302] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96e9f45331d4ce9bba367ae4cdc7700; no voters:
I20260925 12:11:46.067054 2855 leader_election.cc:288] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:46.067099 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:46.067195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2787
I20260925 12:11:46.067224 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [term 1 LEADER]: Becoming Leader. State: Replica: d96e9f45331d4ce9bba367ae4cdc7700, State: Running, Role: LEADER
I20260925 12:11:46.067299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:46.067303 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } }
I20260925 12:11:46.067332 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:46.067485 392 external_mini_cluster.cc:1428] Setting key 2ea9f0a8ed7e81b7587fc9cbba91984a
I20260925 12:11:46.068521 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96e9f45331d4ce9bba367ae4cdc7700. Latest consensus state: current_term: 1 leader_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } } }
I20260925 12:11:46.068616 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:46.068722 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96e9f45331d4ce9bba367ae4cdc7700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37723 } } }
I20260925 12:11:46.068797 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:46.068887 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:46.069437 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:46.070982 2869 catalog_manager.cc:1347] Generated new cluster ID: 22245adbf35046d6a8b0fc4ce054836a
I20260925 12:11:46.071038 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:46.080269 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.067937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.0.98.62:37723)
0925 12:11:46.068282 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:46.068294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:46.068331 (+ 37us) server_negotiation.cc:407] Connection header received
0925 12:11:46.068460 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:46.068469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:46.068572 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:46.069533 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:46.069636 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.070979 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.072045 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.072732 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.072856 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.075584 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:46.075587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:46.075588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:46.075600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:46.078446 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.078872 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.078875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.078876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.078996 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.079250 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.079251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.079251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.079571 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:46.079794 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:46.079864 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 12:11:46.080008 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":91,"threads_started":1}
I20260925 12:11:46.082528 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:46.083244 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:46.113737 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96e9f45331d4ce9bba367ae4cdc7700: Generated new TSK 0
I20260925 12:11:46.113997 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:46.147753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:46.235983 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:46.236178 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:46.236202 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:46.237483 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:46.237524 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:46.237550 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:46.239099 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:46.239332 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:46.239578 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:46.244501 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.244535 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:46.244689 2881 server_base.cc:1034] running on GCE node
W20260925 12:11:46.244557 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:46.244961 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:46.245146 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:46.246317 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790338306246309 us; error 44 us; skew 500 ppm
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:46.249070 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:46.249487 2881 webserver.cc:460] Webserver started at http://127.0.98.1:44485/ using document root <none> and password file <none>
I20260925 12:11:46.249702 2881 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:46.249748 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:46.249871 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:46.250900 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7783406c75f404d806f4cb5be538383"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "57873f948c030e6f6a5f9b667aff4542"
server_key_iv: "342f3d12d16ed8b414226c9e2f2c8562"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.251246 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7783406c75f404d806f4cb5be538383"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "57873f948c030e6f6a5f9b667aff4542"
server_key_iv: "342f3d12d16ed8b414226c9e2f2c8562"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.252620 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:11:46.253376 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.253543 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:46.253643 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "a7783406c75f404d806f4cb5be538383"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "57873f948c030e6f6a5f9b667aff4542"
server_key_iv: "342f3d12d16ed8b414226c9e2f2c8562"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.253726 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:46.319662 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:46.321805 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:46.321971 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:46.322237 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:46.322544 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:46.322598 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.322640 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:46.322665 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.328213 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39051
I20260925 12:11:46.328258 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39051 every 8 connection(s)
I20260925 12:11:46.328634 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:46.333663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 2881
I20260925 12:11:46.333760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:46.333921 392 external_mini_cluster.cc:1428] Setting key 7dad15bea62924454075b14c50d56f68
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:46.339074 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.329396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38873 (local address 127.0.98.62:37723)
0925 12:11:46.329437 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:46.329439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:46.329664 (+ 225us) server_negotiation.cc:407] Connection header received
0925 12:11:46.330584 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:46.330585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:46.330598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:46.330666 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:46.331258 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.331776 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.332836 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.333035 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.333090 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.335752 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:46.335754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:46.335754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:46.335765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:46.337258 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.337838 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.337839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.337840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.337912 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.338316 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.338317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.338317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.338433 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:46.338487 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:46.338843 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 12:11:46.338943 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:11:46.339141 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.329356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37723 (local address 127.0.98.1:38873)
0925 12:11:46.329532 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:46.329574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:46.330492 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:46.330733 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:46.330737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:46.330981 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:46.331168 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.331172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.331860 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.331860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.332766 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.332769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.333106 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.333107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.333198 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.333747 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:46.333756 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:46.335601 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:46.337360 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.337367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.337369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.337720 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.338063 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.338064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.338064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.338229 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.338502 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:46.338510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:46.338682 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 12:11:46.338821 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 12:11:46.338960 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 12:11:46.339993 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37723
I20260925 12:11:46.340106 3010 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:46.340314 3010 heartbeater.cc:511] Master 127.0.98.62:37723 requested a full tablet report, sending...
I20260925 12:11:46.340904 2819 ts_manager.cc:194] Registered new tserver with Master: a7783406c75f404d806f4cb5be538383 (127.0.98.1:39051)
I20260925 12:11:46.341504 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38873
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:46.359901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:46.440115 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:46.440353 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:46.440395 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:46.441682 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:46.441756 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:46.441802 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:46.443399 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:46.443678 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:46.443915 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:46.448812 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.448844 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.448843 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:46.449291 3015 server_base.cc:1034] running on GCE node
I20260925 12:11:46.449544 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:46.449836 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:46.450999 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790338306450989 us; error 33 us; skew 500 ppm
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:46.453896 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:46.454370 3015 webserver.cc:460] Webserver started at http://127.0.98.2:45275/ using document root <none> and password file <none>
I20260925 12:11:46.454615 3015 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:46.454694 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:46.454844 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:46.455945 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "1dafd0c38da34b34800310edc464f3b4"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "ddc568bb286807e930555960fcf248c4"
server_key_iv: "4623e37027320d5ac155088c8eb44f8f"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.456390 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1dafd0c38da34b34800310edc464f3b4"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "ddc568bb286807e930555960fcf248c4"
server_key_iv: "4623e37027320d5ac155088c8eb44f8f"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.457897 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:46.458665 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.458858 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:46.458977 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "1dafd0c38da34b34800310edc464f3b4"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "ddc568bb286807e930555960fcf248c4"
server_key_iv: "4623e37027320d5ac155088c8eb44f8f"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.459056 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:46.477799 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:46.479743 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:46.479915 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:46.480158 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:46.480531 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:46.480597 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.480640 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:46.480667 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.486788 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33295
I20260925 12:11:46.486829 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33295 every 8 connection(s)
I20260925 12:11:46.487203 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:46.494517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3015
I20260925 12:11:46.494592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:46.494800 392 external_mini_cluster.cc:1428] Setting key f7ef429102422dc31a7f734ad6d862ee
I20260925 12:11:46.497027 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.488017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38437 (local address 127.0.98.62:37723)
0925 12:11:46.488083 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:46.488085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:46.488301 (+ 216us) server_negotiation.cc:407] Connection header received
0925 12:11:46.489330 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:46.489332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:46.489342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:46.489411 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:46.490028 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.490534 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.491501 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.491740 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.491798 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.493880 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:46.493881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:46.493881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:46.493888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:46.495362 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.495951 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.495952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.495952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.496002 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.496280 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.496281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.496281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.496379 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:46.496429 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:46.496781 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 12:11:46.496904 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:11:46.497200 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.487979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37723 (local address 127.0.98.2:38437)
0925 12:11:46.488140 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:46.488181 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:46.489202 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:46.489469 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:46.489472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:46.489724 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:46.489918 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.489922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.490623 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.490624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.491411 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.491413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.491819 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.491820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.491918 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.492372 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:46.492381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:46.493794 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:46.495452 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.495458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.495460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.495873 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.496070 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.496071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.496071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.496216 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.496442 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:46.496447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:46.496640 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 12:11:46.496757 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 12:11:46.496945 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260925 12:11:46.497665 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37723
I20260925 12:11:46.497803 3144 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:46.498028 3144 heartbeater.cc:511] Master 127.0.98.62:37723 requested a full tablet report, sending...
I20260925 12:11:46.498406 2819 ts_manager.cc:194] Registered new tserver with Master: 1dafd0c38da34b34800310edc464f3b4 (127.0.98.2:33295)
I20260925 12:11:46.498814 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38437
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:46.519591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:46.601189 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:46.601408 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:46.601433 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:46.603286 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:46.603348 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:46.603387 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:46.604988 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:46.605240 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:46.605490 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:46.610517 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.610532 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:46.610561 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:46.610671 3149 server_base.cc:1034] running on GCE node
I20260925 12:11:46.610960 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:46.611171 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:46.612306 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790338306612297 us; error 31 us; skew 500 ppm
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:46.614882 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:46.615350 3149 webserver.cc:460] Webserver started at http://127.0.98.3:37033/ using document root <none> and password file <none>
I20260925 12:11:46.615571 3149 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:46.615638 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:46.615793 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:46.616930 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "530ca4cf08c149a9b2efebffd132de58"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "9d3a128ebcb194aca027b2f1270cc077"
server_key_iv: "4928a415bfca8406169edd61edab8693"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.617283 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "530ca4cf08c149a9b2efebffd132de58"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "9d3a128ebcb194aca027b2f1270cc077"
server_key_iv: "4928a415bfca8406169edd61edab8693"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.618594 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:46.619417 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.619609 3149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:46.619707 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "530ca4cf08c149a9b2efebffd132de58"
format_stamp: "Formatted at 2026-09-25 12:11:46 on dist-test-slave-2hqq"
server_key: "9d3a128ebcb194aca027b2f1270cc077"
server_key_iv: "4928a415bfca8406169edd61edab8693"
server_key_version: "encryptionkey@0"
I20260925 12:11:46.619827 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:46.651556 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:46.654013 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:46.654179 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:46.654428 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:46.654760 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:46.654817 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.654861 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:46.654891 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:46.660990 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46403
I20260925 12:11:46.661029 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46403 every 8 connection(s)
I20260925 12:11:46.661366 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:46.664875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3149
I20260925 12:11:46.664973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:46.665159 392 external_mini_cluster.cc:1428] Setting key b71038a4969bbe868a0d98db0d26ea5d
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:46.671393 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.662157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41583 (local address 127.0.98.62:37723)
0925 12:11:46.662203 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:46.662205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:46.662440 (+ 235us) server_negotiation.cc:407] Connection header received
0925 12:11:46.663468 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:46.663470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:46.663481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:46.663546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:46.664125 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.664707 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.665940 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.666216 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.666272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.668626 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:46.668627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:46.668628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:46.668635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:46.669998 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.670412 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.670413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.670413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.670476 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:46.670737 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:46.670738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:46.670738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:46.670857 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:46.670931 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:46.671210 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 12:11:46.671299 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:11:46.671586 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:46.662176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37723 (local address 127.0.98.3:41583)
0925 12:11:46.662326 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:46.662363 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:46.663370 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:46.663603 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:46.663606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:46.663863 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:46.664052 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.664056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.664795 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.664796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.665875 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:46.665877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:46.666292 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:46.666293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:46.666391 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:46.666943 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:46.666952 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:46.668489 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:46.670096 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.670103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.670105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.670335 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.670546 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:46.670546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:46.670547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:46.670664 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:46.670923 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:46.670929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:46.671076 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 12:11:46.671188 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 12:11:46.671347 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 12:11:46.672111 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37723
I20260925 12:11:46.672194 3278 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:46.672344 3278 heartbeater.cc:511] Master 127.0.98.62:37723 requested a full tablet report, sending...
I20260925 12:11:46.672706 2819 ts_manager.cc:194] Registered new tserver with Master: 530ca4cf08c149a9b2efebffd132de58 (127.0.98.3:46403)
I20260925 12:11:46.673059 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41583
I20260925 12:11:46.677018 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 12:11:46.688136 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:46.687133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.98.62:37723)
0925 12:11:46.687193 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:46.687195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:46.687367 (+ 172us) server_negotiation.cc:407] Connection header received
0925 12:11:46.687399 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:46.687400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:46.687411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:46.687458 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:46.688031 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54222: BlockingRecv error: recv got EOF from 127.0.0.1:54222 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:11:46.689163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2881
I20260925 12:11:46.693457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3015
I20260925 12:11:46.697765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3149
I20260925 12:11:46.702019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 2787
2026-09-25T12:11:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1934 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[3290](info): set up 2 sockets
Sep 25 12:11:46 dist-test-slave-2hqq krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260925 12:11:47.736459 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:47 dist-test-slave-2hqq krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:47.855831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:47.935990 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:47.936192 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:47.936213 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:47.937384 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:47.937427 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:47.937441 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:47.937454 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:47.937465 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:47.939224 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:47.939427 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:47.939677 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:47.944473 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:47.944487 3303 server_base.cc:1034] running on GCE node
W20260925 12:11:47.944460 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:47.944459 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:47.944953 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:47.945205 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:47.946350 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790338307946324 us; error 38 us; skew 500 ppm
Sep 25 12:11:47 dist-test-slave-2hqq krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:47.948800 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:47.949272 3303 webserver.cc:460] Webserver started at http://127.0.98.62:39599/ using document root <none> and password file <none>
I20260925 12:11:47.949515 3303 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:47.949566 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:47.949724 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:47.950800 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "3f3072d6032e45bfa4f529e74602b303"
format_stamp: "Formatted at 2026-09-25 12:11:47 on dist-test-slave-2hqq"
server_key: "3b9c98ad4b101e320d46cf30b89cc0aa"
server_key_iv: "1e07a6f3419b39336ccc6aef29e6c706"
server_key_version: "encryptionkey@0"
I20260925 12:11:47.951155 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f3072d6032e45bfa4f529e74602b303"
format_stamp: "Formatted at 2026-09-25 12:11:47 on dist-test-slave-2hqq"
server_key: "3b9c98ad4b101e320d46cf30b89cc0aa"
server_key_iv: "1e07a6f3419b39336ccc6aef29e6c706"
server_key_version: "encryptionkey@0"
I20260925 12:11:47.952455 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:47.953341 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:47.953593 3303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:47.953697 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "3f3072d6032e45bfa4f529e74602b303"
format_stamp: "Formatted at 2026-09-25 12:11:47 on dist-test-slave-2hqq"
server_key: "3b9c98ad4b101e320d46cf30b89cc0aa"
server_key_iv: "1e07a6f3419b39336ccc6aef29e6c706"
server_key_version: "encryptionkey@0"
I20260925 12:11:47.953778 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:47.978528 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:47.980858 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:47.981024 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:47.984977 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42523
I20260925 12:11:47.985016 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42523 every 8 connection(s)
I20260925 12:11:47.985385 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:47.986232 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:47.988603 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303: Bootstrap starting.
I20260925 12:11:47.989187 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:47.989485 3371 log.cc:824] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:47.990154 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303: No bootstrap required, opened a new log
I20260925 12:11:47.990655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3303
I20260925 12:11:47.990772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:47.990944 392 external_mini_cluster.cc:1428] Setting key 11b6b287613a3418276ce51a92b6ea80
I20260925 12:11:47.992887 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } }
I20260925 12:11:47.993058 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:47.993096 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3072d6032e45bfa4f529e74602b303, State: Initialized, Role: FOLLOWER
I20260925 12:11:47.993256 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } }
I20260925 12:11:47.993328 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:47.993348 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:47.993415 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:47.994024 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } }
I20260925 12:11:47.994127 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3072d6032e45bfa4f529e74602b303; no voters:
I20260925 12:11:47.994274 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:47.994339 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:47.994484 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3072d6032e45bfa4f529e74602b303, State: Running, Role: LEADER
I20260925 12:11:47.994506 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:47.994621 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } }
I20260925 12:11:47.994958 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f3072d6032e45bfa4f529e74602b303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } } }
I20260925 12:11:47.994948 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3072d6032e45bfa4f529e74602b303. Latest consensus state: current_term: 1 leader_uuid: "3f3072d6032e45bfa4f529e74602b303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3072d6032e45bfa4f529e74602b303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42523 } } }
I20260925 12:11:47.995111 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:47.995119 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:47.995890 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:47.996431 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:47.998306 3384 catalog_manager.cc:1347] Generated new cluster ID: 8d2fcc6b22d84f8e97c32cadce0345cb
I20260925 12:11:47.998363 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:47 dist-test-slave-2hqq krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:48.003899 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:47.991492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.0.98.62:42523)
0925 12:11:47.991701 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:47.991711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:47.991747 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:11:47.992848 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:47.992857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:47.992924 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:47.993087 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:47.993835 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:47.995271 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:47.996439 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:47.997033 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:47.997150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:47.999999 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:48.000003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:48.000003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:48.000014 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:48.002219 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:48.002669 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:48.002672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:48.002672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:48.002789 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:48.003012 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:48.003012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:48.003013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:48.003299 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:48.003426 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:48.003577 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 12:11:48.003706 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260925 12:11:48.005061 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:11:48.006011 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:48.006752 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:48.014053 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3072d6032e45bfa4f529e74602b303: Generated new TSK 0
I20260925 12:11:48.014236 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:11:48.023313 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:48.019287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.0.98.62:42523)
0925 12:11:48.019351 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:48.019353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:48.019454 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:11:48.019549 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:48.019550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:48.019566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:48.019663 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:48.020017 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:48.020859 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:48.021651 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:48.022002 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:48.022083 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:48.022827 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:48.022828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:48.022828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:48.023061 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:48.023081 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:48.023088 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:48.023150 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:48.023202 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 12:11:48.023229 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:11:48.025638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3303
2026-09-25T12:11:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1321 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:48 dist-test-slave-2hqq krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:48 dist-test-slave-2hqq krb5kdc[3404](info): set up 2 sockets
Sep 25 12:11:48 dist-test-slave-2hqq krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260925 12:11:49.104285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:49.221246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:49.301829 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:49.302071 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:49.302107 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:49.303246 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:49.303310 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:49.303333 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:49.303362 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:49.303393 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:49.304877 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:49.305146 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:49.305377 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:49.310600 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.310600 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.310650 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.310935 3417 server_base.cc:1034] running on GCE node
I20260925 12:11:49.311211 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:49.311503 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:49.312633 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790338309312619 us; error 34 us; skew 500 ppm
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:49.315060 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:49.315542 3417 webserver.cc:460] Webserver started at http://127.0.98.62:44495/ using document root <none> and password file <none>
I20260925 12:11:49.315794 3417 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:49.315862 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:49.316004 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:49.317111 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "44ab7322e80248d7b4f2fd3cb30c3425"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "eec4dd70a9a41029042d65148470facc"
server_key_iv: "0071c32495c224f159461f1e10bb32e4"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.317463 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "44ab7322e80248d7b4f2fd3cb30c3425"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "eec4dd70a9a41029042d65148470facc"
server_key_iv: "0071c32495c224f159461f1e10bb32e4"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.318727 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:49.319442 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.319617 3417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:49.319718 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "44ab7322e80248d7b4f2fd3cb30c3425"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "eec4dd70a9a41029042d65148470facc"
server_key_iv: "0071c32495c224f159461f1e10bb32e4"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.319799 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:49.336864 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:49.338647 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:49.338811 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:49.342759 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44353
I20260925 12:11:49.342813 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44353 every 8 connection(s)
I20260925 12:11:49.343190 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:49.344188 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:49.345867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3417
I20260925 12:11:49.345978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:49.346143 392 external_mini_cluster.cc:1428] Setting key c4eef75a838e3a032e074f3eae5ad0e6
I20260925 12:11:49.346735 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425: Bootstrap starting.
I20260925 12:11:49.347460 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:49.347771 3485 log.cc:824] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:49.348546 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425: No bootstrap required, opened a new log
I20260925 12:11:49.350740 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260925 12:11:49.350900 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:49.350934 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ab7322e80248d7b4f2fd3cb30c3425, State: Initialized, Role: FOLLOWER
I20260925 12:11:49.351063 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260925 12:11:49.351161 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:49.351193 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:49.351281 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:49.351815 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260925 12:11:49.351961 3485 leader_election.cc:302] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ab7322e80248d7b4f2fd3cb30c3425; no voters:
I20260925 12:11:49.352110 3485 leader_election.cc:288] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:49.352130 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:49.352236 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [term 1 LEADER]: Becoming Leader. State: Replica: 44ab7322e80248d7b4f2fd3cb30c3425, State: Running, Role: LEADER
I20260925 12:11:49.352358 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:49.352365 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } }
I20260925 12:11:49.352790 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ab7322e80248d7b4f2fd3cb30c3425. Latest consensus state: current_term: 1 leader_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } } }
I20260925 12:11:49.352962 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:49.352802 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ab7322e80248d7b4f2fd3cb30c3425" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44353 } } }
I20260925 12:11:49.353263 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:49.353260 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:49.353811 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:49.355402 3499 catalog_manager.cc:1347] Generated new cluster ID: f7f4c37f4e9b49de9c78e130cb04795a
I20260925 12:11:49.355496 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:49.357436 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.346667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.0.98.62:44353)
0925 12:11:49.346861 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:49.346886 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:49.346924 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:11:49.347516 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:49.347525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:49.347591 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:49.347751 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:49.348349 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.349726 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.350463 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.351070 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.351172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.354232 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:49.354236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:49.354236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:49.354247 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:49.355941 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.356303 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.356306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.356306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.356401 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.356627 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.356628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.356628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.356858 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:49.357032 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:49.357132 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 12:11:49.357268 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260925 12:11:49.365550 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:49.366099 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:49.369987 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ab7322e80248d7b4f2fd3cb30c3425: Generated new TSK 0
I20260925 12:11:49.370155 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:49.399808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:49.482386 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:49.482601 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:49.482625 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:49.483876 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:49.483929 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:49.483956 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:49.485483 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:49.485723 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:49.485975 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:49.491102 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.491115 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.491132 3511 server_base.cc:1034] running on GCE node
W20260925 12:11:49.491102 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.491593 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:49.491798 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:49.492980 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790338309492962 us; error 38 us; skew 500 ppm
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:49.495707 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:49.496163 3511 webserver.cc:460] Webserver started at http://127.0.98.1:35313/ using document root <none> and password file <none>
I20260925 12:11:49.496362 3511 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:49.496403 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:49.496552 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:49.497588 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "602fe3fe41624151bc67c689ffd8dd48"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "b820267ebed0cdd2b772a4a08de5cd70"
server_key_iv: "6bf12764d6f02fc0eb49cc7a5eda0d11"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.497939 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "602fe3fe41624151bc67c689ffd8dd48"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "b820267ebed0cdd2b772a4a08de5cd70"
server_key_iv: "6bf12764d6f02fc0eb49cc7a5eda0d11"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.499213 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:11:49.499989 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.500166 3511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:49.500243 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "602fe3fe41624151bc67c689ffd8dd48"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "b820267ebed0cdd2b772a4a08de5cd70"
server_key_iv: "6bf12764d6f02fc0eb49cc7a5eda0d11"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.500295 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:49.509548 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:49.511801 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:49.511919 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:49.512091 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:49.512403 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:49.512437 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.512480 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:49.512548 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.518011 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46351
I20260925 12:11:49.518150 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46351 every 8 connection(s)
I20260925 12:11:49.518415 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:49.524600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3511
I20260925 12:11:49.524719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:49.524971 392 external_mini_cluster.cc:1428] Setting key 920a0c5494fae7f89d588e8aa7cfe75a
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:49.529172 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.519248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34783 (local address 127.0.98.62:44353)
0925 12:11:49.519323 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:49.519326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:49.519534 (+ 208us) server_negotiation.cc:407] Connection header received
0925 12:11:49.520084 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:49.520085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:49.520095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:49.520176 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:49.521279 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.521791 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.523043 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.523248 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.523314 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.525882 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:49.525884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:49.525884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:49.525893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:49.527651 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.528096 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.528098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.528098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.528164 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.528403 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.528404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.528404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.528588 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:49.528660 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:49.528930 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 12:11:49.529029 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:11:49.529199 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.519204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.1:34783)
0925 12:11:49.519393 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:49.519446 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:49.519977 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:49.520290 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:49.520294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:49.520978 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:49.521176 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.521182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.521881 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.521882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.522931 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.522934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.523365 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.523366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.523587 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.524091 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:49.524100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:49.525782 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:49.527744 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.527750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.527752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.528019 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.528234 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.528235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.528235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.528339 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.528662 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:49.528667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:49.528792 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 12:11:49.528881 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 12:11:49.529030 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":124,"threads_started":1}
I20260925 12:11:49.529897 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260925 12:11:49.530037 3640 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:49.530249 3640 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260925 12:11:49.530814 3449 ts_manager.cc:194] Registered new tserver with Master: 602fe3fe41624151bc67c689ffd8dd48 (127.0.98.1:46351)
I20260925 12:11:49.531440 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34783
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:49.550716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:49.633256 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:49.633479 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:49.633502 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:49.634687 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:49.634732 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:49.634758 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:49.636188 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:49.636440 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:49.636696 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:11:49.642069 3645 server_base.cc:1034] running on GCE node
W20260925 12:11:49.642067 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.642081 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.642086 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.642448 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:49.642714 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:49.643860 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790338309643845 us; error 27 us; skew 500 ppm
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:49.646584 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:49.647078 3645 webserver.cc:460] Webserver started at http://127.0.98.2:38497/ using document root <none> and password file <none>
I20260925 12:11:49.647323 3645 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:49.647406 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:49.647572 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:49.648816 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "62d7d06827a645abb1a13584adf58654"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "fa67783f6953af649d854141e880f420"
server_key_iv: "b33ed76c232d1902d71772eebdcee189"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.649210 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62d7d06827a645abb1a13584adf58654"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "fa67783f6953af649d854141e880f420"
server_key_iv: "b33ed76c232d1902d71772eebdcee189"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.650624 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:49.651466 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.651715 3645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:49.651834 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "62d7d06827a645abb1a13584adf58654"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "fa67783f6953af649d854141e880f420"
server_key_iv: "b33ed76c232d1902d71772eebdcee189"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.651927 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:49.682668 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:49.684937 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:49.685120 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:49.685396 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:49.685746 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:49.685806 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.685851 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:49.685876 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.692608 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36517
I20260925 12:11:49.692678 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36517 every 8 connection(s)
I20260925 12:11:49.693041 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:49.696463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3645
I20260925 12:11:49.696591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:49.696844 392 external_mini_cluster.cc:1428] Setting key d04d52154379854eb7af6b6bc2aade0a
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:49.704178 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.693879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53839 (local address 127.0.98.62:44353)
0925 12:11:49.693948 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:49.693950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:49.694187 (+ 237us) server_negotiation.cc:407] Connection header received
0925 12:11:49.694786 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:49.694787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:49.694801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:49.694857 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:49.695901 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.696539 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.698067 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.698265 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.698333 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.700951 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:49.700953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:49.700953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:49.700962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:49.702586 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.703121 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.703122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.703122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.703229 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.703491 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.703492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.703492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.703617 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:49.703695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:49.703983 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 12:11:49.704089 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:11:49.704362 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.693859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.2:53839)
0925 12:11:49.694047 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:49.694085 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:49.694656 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:49.694934 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:49.694939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:49.695577 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:49.695788 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.695793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.696666 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.696668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.697967 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.697969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.698357 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.698359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.698507 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.699080 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:49.699093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:49.700849 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:49.702717 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.702724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.702727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.703030 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.703303 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.703304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.703304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.703420 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.703685 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:49.703692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:49.703845 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 12:11:49.703961 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 12:11:49.704124 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260925 12:11:49.704953 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260925 12:11:49.705075 3774 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:49.705255 3774 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260925 12:11:49.705682 3449 ts_manager.cc:194] Registered new tserver with Master: 62d7d06827a645abb1a13584adf58654 (127.0.98.2:36517)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:11:49.706269 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:49.723671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:49.806442 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:49.806674 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:49.806701 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:49.808023 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:49.808076 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:49.808108 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:49.809774 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:49.810065 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:49.810324 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:49.815663 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.815714 3779 server_base.cc:1034] running on GCE node
W20260925 12:11:49.815660 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:49.815719 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:49.816082 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:49.816298 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:49.817471 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790338309817449 us; error 54 us; skew 500 ppm
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:49.820098 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:49.820603 3779 webserver.cc:460] Webserver started at http://127.0.98.3:38435/ using document root <none> and password file <none>
I20260925 12:11:49.820839 3779 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:49.820907 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:49.821074 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:49.822211 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "df8bb0e1c062420f861dda4b06a2bb41"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "54e0d07182fe6b983b9f1d924cbd7440"
server_key_iv: "9f9ac0b3481763b1df9bed30eaf8433b"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.822620 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df8bb0e1c062420f861dda4b06a2bb41"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "54e0d07182fe6b983b9f1d924cbd7440"
server_key_iv: "9f9ac0b3481763b1df9bed30eaf8433b"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.824051 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:49.824918 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.825102 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:11:49.825201 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "df8bb0e1c062420f861dda4b06a2bb41"
format_stamp: "Formatted at 2026-09-25 12:11:49 on dist-test-slave-2hqq"
server_key: "54e0d07182fe6b983b9f1d924cbd7440"
server_key_iv: "9f9ac0b3481763b1df9bed30eaf8433b"
server_key_version: "encryptionkey@0"
I20260925 12:11:49.825284 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:49.842470 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:49.844786 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:49.844956 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:49.845196 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:49.845506 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:49.845561 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.845602 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:49.845629 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:49.851584 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43783
I20260925 12:11:49.851626 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43783 every 8 connection(s)
I20260925 12:11:49.851994 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:49.858793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3779
I20260925 12:11:49.858919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:49.859203 392 external_mini_cluster.cc:1428] Setting key 7ecafa5ba8d441b211b537b866975e6a
I20260925 12:11:49.861642 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.852825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48013 (local address 127.0.98.62:44353)
0925 12:11:49.852885 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:49.852887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:49.853116 (+ 229us) server_negotiation.cc:407] Connection header received
0925 12:11:49.853656 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:49.853658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:49.853670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:49.853727 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:49.854734 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.855237 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.856266 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.856534 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.856595 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.858670 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:49.858672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:49.858673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:49.858681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:49.860408 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.860813 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.860814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.860814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.860879 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.861108 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.861109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.861109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.861221 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:49.861277 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:49.861512 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 12:11:49.861576 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:11:49.861794 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.852827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44353 (local address 127.0.98.3:48013)
0925 12:11:49.852995 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:49.853031 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:49.853551 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:49.853799 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:49.853804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:49.854432 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:49.854625 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.854629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.855354 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.855355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.856159 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:49.856161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.856600 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.856601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:49.856736 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.857192 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:49.857200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:49.858575 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:49.860481 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.860523 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.860525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.860747 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.860948 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:49.860949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:49.860949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:49.861050 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:49.861276 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:49.861280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:49.861394 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 12:11:49.861480 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 12:11:49.861598 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260925 12:11:49.862286 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44353
I20260925 12:11:49.862391 3908 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:49.862564 3908 heartbeater.cc:511] Master 127.0.98.62:44353 requested a full tablet report, sending...
I20260925 12:11:49.862936 3449 ts_manager.cc:194] Registered new tserver with Master: df8bb0e1c062420f861dda4b06a2bb41 (127.0.98.3:43783)
I20260925 12:11:49.863330 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48013
I20260925 12:11:49.871176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:49.894328 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:49.886720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.0.98.62:44353)
0925 12:11:49.886784 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:49.886788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:49.886823 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:11:49.886895 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:49.886896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:49.886905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:49.886967 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:49.887665 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.888204 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.889111 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:49.889299 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:49.889366 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:49.891763 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:49.891764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:49.891764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:49.891771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:49.893180 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.893553 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.893554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.893555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.893606 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:49.893824 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:49.893824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:49.893825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:49.893990 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:49.894068 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:49.894194 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 12:11:49.894260 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 12:11:49.894634 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48632
I20260925 12:11:49.895105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3511
I20260925 12:11:49.899264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3645
I20260925 12:11:49.903440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3779
I20260925 12:11:49.907550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3417
2026-09-25T12:11:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1883 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3920](info): set up 2 sockets
Sep 25 12:11:49 dist-test-slave-2hqq krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260925 12:11:50.940582 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:51.064481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35873
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:51.147461 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:51.147696 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:51.147733 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:51.148923 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:51.148998 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:51.149024 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:51.149049 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:51.149080 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:51.150520 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:51.150797 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:51.151026 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:51.156038 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:51.156055 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:51.156095 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.156360 3933 server_base.cc:1034] running on GCE node
I20260925 12:11:51.156641 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:51.156896 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:51.158046 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790338311158038 us; error 35 us; skew 500 ppm
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:51.160634 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:51.161072 3933 webserver.cc:460] Webserver started at http://127.0.98.62:38063/ using document root <none> and password file <none>
I20260925 12:11:51.161321 3933 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:51.161389 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:51.161535 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:51.162617 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "b4834dfbe8304a74b94f121cdfb8530d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "d5f1a2900c164ca22193f2f8313fbd98"
server_key_iv: "cb7a72a13e7af2c6c88e51b3822c6204"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.162984 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4834dfbe8304a74b94f121cdfb8530d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "d5f1a2900c164ca22193f2f8313fbd98"
server_key_iv: "cb7a72a13e7af2c6c88e51b3822c6204"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.164340 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:51.165163 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.165386 3933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:51.165478 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "b4834dfbe8304a74b94f121cdfb8530d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "d5f1a2900c164ca22193f2f8313fbd98"
server_key_iv: "cb7a72a13e7af2c6c88e51b3822c6204"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.165566 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:51.174677 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:51.176579 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:51.176739 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:51.180562 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35873
I20260925 12:11:51.180568 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35873 every 8 connection(s)
I20260925 12:11:51.180968 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:51.181850 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:51.183990 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: Bootstrap starting.
I20260925 12:11:51.184574 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:51.184825 4001 log.cc:824] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:51.185513 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: No bootstrap required, opened a new log
I20260925 12:11:51.187045 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } }
I20260925 12:11:51.187186 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:51.187208 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4834dfbe8304a74b94f121cdfb8530d, State: Initialized, Role: FOLLOWER
I20260925 12:11:51.187367 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } }
I20260925 12:11:51.187451 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:51.187474 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:51.187503 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:51.187994 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } }
I20260925 12:11:51.188074 4001 leader_election.cc:302] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4834dfbe8304a74b94f121cdfb8530d; no voters:
I20260925 12:11:51.188194 4001 leader_election.cc:288] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:51.188334 4004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:51.188534 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:51.188750 4004 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [term 1 LEADER]: Becoming Leader. State: Replica: b4834dfbe8304a74b94f121cdfb8530d, State: Running, Role: LEADER
I20260925 12:11:51.188910 4004 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } }
I20260925 12:11:51.189430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 3933
I20260925 12:11:51.189545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:51.189754 392 external_mini_cluster.cc:1428] Setting key ffdb88ba263c66880bb9d8d21b1597b2
I20260925 12:11:51.190191 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4834dfbe8304a74b94f121cdfb8530d. Latest consensus state: current_term: 1 leader_uuid: "b4834dfbe8304a74b94f121cdfb8530d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } } }
I20260925 12:11:51.190482 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:51.190220 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4834dfbe8304a74b94f121cdfb8530d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4834dfbe8304a74b94f121cdfb8530d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35873 } } }
W20260925 12:11:51.190622 4020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 12:11:51.190673 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d [sys.catalog]: This master's current role is: LEADER
W20260925 12:11:51.190795 4020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:11:51.190850 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:51.191038 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:51.192677 4024 catalog_manager.cc:1347] Generated new cluster ID: b5adb473727649609ff7e020894b4472
I20260925 12:11:51.192731 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:51.200224 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.190424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.98.62:35873)
0925 12:11:51.190949 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.190959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.191001 (+ 42us) server_negotiation.cc:407] Connection header received
0925 12:11:51.191137 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.191144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.191205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.191355 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.192153 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.193329 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.194108 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.194619 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.194739 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.197114 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.197118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.197119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.197130 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.198645 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.199012 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.199016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.199016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.199109 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.199352 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.199353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.199353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.199604 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.199798 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.199865 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.200008 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":82,"threads_started":1}
I20260925 12:11:51.218050 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:51.218649 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:51.227094 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4834dfbe8304a74b94f121cdfb8530d: Generated new TSK 0
I20260925 12:11:51.227341 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:51.255416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:51.359458 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:51.359651 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:51.359674 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:51.360879 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:51.360919 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:51.360945 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:51.362330 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:51.362509 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:51.362686 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:51.367502 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.367580 4028 server_base.cc:1034] running on GCE node
W20260925 12:11:51.367511 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:51.367517 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.367990 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:51.368323 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:51.369463 4028 hybrid_clock.cc:648] HybridClock initialized: now 1790338311369421 us; error 58 us; skew 500 ppm
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:51.371882 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:51.372376 4028 webserver.cc:460] Webserver started at http://127.0.98.1:36113/ using document root <none> and password file <none>
I20260925 12:11:51.372690 4028 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:51.372757 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:51.372886 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:51.373885 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "1953f51de6634e8baaa1c1e96a61639b"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "b20a294d0be7bfe4d3749507a9542fe8"
server_key_iv: "d13621549be27bffb5c4b26d1ab5e513"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.374235 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1953f51de6634e8baaa1c1e96a61639b"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "b20a294d0be7bfe4d3749507a9542fe8"
server_key_iv: "d13621549be27bffb5c4b26d1ab5e513"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.375499 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:51.376214 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.376397 4028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:51.376546 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "1953f51de6634e8baaa1c1e96a61639b"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "b20a294d0be7bfe4d3749507a9542fe8"
server_key_iv: "d13621549be27bffb5c4b26d1ab5e513"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.376622 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:51.394870 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:51.396842 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:51.397019 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:51.397276 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:51.397595 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:51.397650 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.397699 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:51.397747 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.403612 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37101
I20260925 12:11:51.403661 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37101 every 8 connection(s)
I20260925 12:11:51.403982 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:51.410943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4028
I20260925 12:11:51.411059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:51.411392 392 external_mini_cluster.cc:1428] Setting key 9820036721cd95cef95ebf2d837e05c2
I20260925 12:11:51.413623 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.404777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53209 (local address 127.0.98.62:35873)
0925 12:11:51.404839 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.404841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.405016 (+ 175us) server_negotiation.cc:407] Connection header received
0925 12:11:51.405931 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.405932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.405941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.406001 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.406589 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.407086 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.408027 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.408218 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.408265 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.410355 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.410357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.410357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.410365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.411868 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.412405 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.412406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.412406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.412474 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.412818 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.412818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.412819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.412929 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.412986 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.413376 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.413482 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:11:51.413724 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.404720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35873 (local address 127.0.98.1:53209)
0925 12:11:51.404880 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:51.404920 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:51.405842 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:51.406070 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:51.406074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:51.406328 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:51.406515 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.406519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.407162 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.407163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.407954 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.407955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.408286 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.408287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.408392 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.408907 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:51.408915 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:51.410271 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:51.411966 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.411973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.411975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.412313 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.412603 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.412604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.412604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.412746 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.412998 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:51.413003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:51.413168 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 12:11:51.413292 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 12:11:51.413484 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260925 12:11:51.414181 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35873
I20260925 12:11:51.414283 4157 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:51.414489 4157 heartbeater.cc:511] Master 127.0.98.62:35873 requested a full tablet report, sending...
I20260925 12:11:51.414997 3965 ts_manager.cc:194] Registered new tserver with Master: 1953f51de6634e8baaa1c1e96a61639b (127.0.98.1:37101)
I20260925 12:11:51.415727 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53209
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:51.435695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:51.516023 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:51.516247 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:51.516271 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:51.517382 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:51.517418 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:51.517443 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:51.518865 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:51.519048 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:51.519249 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:51.524389 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:51.524401 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.524433 4162 server_base.cc:1034] running on GCE node
W20260925 12:11:51.524406 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.524848 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:51.525100 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:51.526290 4162 hybrid_clock.cc:648] HybridClock initialized: now 1790338311526292 us; error 42 us; skew 500 ppm
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:51.528795 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:51.529229 4162 webserver.cc:460] Webserver started at http://127.0.98.2:45545/ using document root <none> and password file <none>
I20260925 12:11:51.529455 4162 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:51.529520 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:51.529670 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:51.530865 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "19ae2660557e443da928feb8beafad4d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "c45e5cccce1388500b8ac2de915497f9"
server_key_iv: "944f0f53b41cc3afb7745da989be9781"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.531226 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "19ae2660557e443da928feb8beafad4d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "c45e5cccce1388500b8ac2de915497f9"
server_key_iv: "944f0f53b41cc3afb7745da989be9781"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.532598 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:11:51.533360 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.533542 4162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:51.533638 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "19ae2660557e443da928feb8beafad4d"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "c45e5cccce1388500b8ac2de915497f9"
server_key_iv: "944f0f53b41cc3afb7745da989be9781"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.533740 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:51.559465 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:51.561564 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:51.561736 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:51.561990 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:51.562294 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:51.562345 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.562386 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:51.562417 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.568152 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34663
I20260925 12:11:51.568215 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34663 every 8 connection(s)
I20260925 12:11:51.568621 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:51.570482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4162
I20260925 12:11:51.570600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:51.570809 392 external_mini_cluster.cc:1428] Setting key ee7476e6e439a27a21a0e8f4bb7ebdd3
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:51.578253 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.569363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49381 (local address 127.0.98.62:35873)
0925 12:11:51.569433 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.569435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.569631 (+ 196us) server_negotiation.cc:407] Connection header received
0925 12:11:51.570630 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.570631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.570645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.570747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.571498 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.571989 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.573098 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.573309 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.573396 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.575460 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.575462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.575462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.575470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.576879 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.577292 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.577293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.577294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.577353 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.577604 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.577605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.577605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.577720 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.577795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.578071 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.578152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:11:51.578431 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.569356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35873 (local address 127.0.98.2:49381)
0925 12:11:51.569527 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:51.569562 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:51.570479 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:51.570811 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:51.570814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:51.571175 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:51.571424 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.571431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.572088 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.572089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.573009 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.573012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.573405 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.573406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.573538 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.574076 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:51.574085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:51.575368 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:51.576974 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.576981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.576983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.577211 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.577428 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.577428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.577429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.577538 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.577786 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:51.577792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:51.577936 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 12:11:51.578041 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 12:11:51.578198 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260925 12:11:51.578902 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35873
I20260925 12:11:51.579008 4291 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:51.579248 4291 heartbeater.cc:511] Master 127.0.98.62:35873 requested a full tablet report, sending...
I20260925 12:11:51.579695 3965 ts_manager.cc:194] Registered new tserver with Master: 19ae2660557e443da928feb8beafad4d (127.0.98.2:34663)
I20260925 12:11:51.580031 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:51.596035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:51.676352 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:51.676630 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:51.676669 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:51.677886 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:51.677945 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:51.677989 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:51.679536 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:51.679797 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:51.680014 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:51.684950 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.684965 4296 server_base.cc:1034] running on GCE node
W20260925 12:11:51.684949 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:51.684950 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:51.685372 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:51.685571 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:51.686729 4296 hybrid_clock.cc:648] HybridClock initialized: now 1790338311686726 us; error 33 us; skew 500 ppm
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:51.689261 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:51.689765 4296 webserver.cc:460] Webserver started at http://127.0.98.3:44287/ using document root <none> and password file <none>
I20260925 12:11:51.690056 4296 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:51.690140 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:51.690279 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:51.691316 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a8401214b834ab2852aa18b68f912c2"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "3a87757b40e158027d2901a74a28f6e0"
server_key_iv: "6ea23ad6ee8a2bdc9bb0b3da1fc6ca3f"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.691681 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8401214b834ab2852aa18b68f912c2"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "3a87757b40e158027d2901a74a28f6e0"
server_key_iv: "6ea23ad6ee8a2bdc9bb0b3da1fc6ca3f"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.693086 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:51.693840 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.694005 4296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:51.694101 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "9a8401214b834ab2852aa18b68f912c2"
format_stamp: "Formatted at 2026-09-25 12:11:51 on dist-test-slave-2hqq"
server_key: "3a87757b40e158027d2901a74a28f6e0"
server_key_iv: "6ea23ad6ee8a2bdc9bb0b3da1fc6ca3f"
server_key_version: "encryptionkey@0"
I20260925 12:11:51.694182 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:51.716697 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:51.718961 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:51.719161 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:51.719419 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:51.719749 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:51.719810 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.719851 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:51.719882 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:51.725874 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46401
I20260925 12:11:51.725921 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46401 every 8 connection(s)
I20260925 12:11:51.726295 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:51.730784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4296
I20260925 12:11:51.730885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:51.731189 392 external_mini_cluster.cc:1428] Setting key 10ad5f516acb722857032b8d6002dcca
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:51.735980 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.727114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44591 (local address 127.0.98.62:35873)
0925 12:11:51.727180 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.727182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.727420 (+ 238us) server_negotiation.cc:407] Connection header received
0925 12:11:51.728518 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.728520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.728533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.728591 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.729241 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.729760 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.730912 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.731188 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.731285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.733572 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.733574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.733574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.733592 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.734745 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.735153 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.735154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.735155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.735214 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.735472 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.735473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.735473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.735575 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.735625 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.735857 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.735918 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:11:51.736210 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.727018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35873 (local address 127.0.98.3:44591)
0925 12:11:51.727275 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:51.727321 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:51.728379 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:51.728667 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:51.728671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:51.728931 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:51.729120 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.729124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.729852 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.729853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.730761 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:51.730764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.731304 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.731305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:51.731430 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.732052 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:51.732062 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:51.733490 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:51.734833 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.734838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.734840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.735057 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.735301 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:51.735302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:51.735303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:51.735408 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:51.735633 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:51.735637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:51.735748 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 12:11:51.735831 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:11:51.735947 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260925 12:11:51.736707 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35873
I20260925 12:11:51.736804 4425 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:51.737016 4425 heartbeater.cc:511] Master 127.0.98.62:35873 requested a full tablet report, sending...
I20260925 12:11:51.737376 3965 ts_manager.cc:194] Registered new tserver with Master: 9a8401214b834ab2852aa18b68f912c2 (127.0.98.3:46401)
I20260925 12:11:51.737843 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44591
I20260925 12:11:51.743019 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:51.765937 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.757720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.98.1:37101)
0925 12:11:51.757943 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.757948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.757959 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:51.757985 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.757987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.758001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.758075 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.758667 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.759402 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.760135 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.760465 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.760606 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.762834 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.762841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.762841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.762847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.764547 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.764899 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.764903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.764903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.764963 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.765150 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.765150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.765151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.765448 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.765569 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.765655 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.765743 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":116,"threads_started":1}
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:51.773916 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:51.767233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.0.98.62:35873)
0925 12:11:51.767303 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:51.767304 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:51.767312 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:51.767351 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:51.767352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:51.767360 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:51.767406 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:51.767835 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.768392 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.769122 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:51.769360 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:51.769421 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:51.771556 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:51.771558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:51.771558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:51.771565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:51.772974 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.773298 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.773299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.773299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.773347 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:51.773508 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:51.773508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:51.773508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:51.773639 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:51.773710 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:51.773799 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 12:11:51.773853 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 12:11:51.774169 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46878
I20260925 12:11:51.774552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4028
I20260925 12:11:51.779245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4162
I20260925 12:11:51.783615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4296
I20260925 12:11:51.787812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 3933
2026-09-25T12:11:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1880 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[4437](info): set up 2 sockets
Sep 25 12:11:51 dist-test-slave-2hqq krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260925 12:11:52.824762 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:52 dist-test-slave-2hqq krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:52.944231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:53.025105 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:53.025357 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:53.025394 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:53.026544 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:53.026618 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:53.026641 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:53.026670 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:53.026700 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:53.028174 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:53.028443 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:53.028697 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:53.033355 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.033389 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.033386 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:53.033550 4450 server_base.cc:1034] running on GCE node
I20260925 12:11:53.033789 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:53.034018 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:53.035281 4450 hybrid_clock.cc:648] HybridClock initialized: now 1790338313035263 us; error 35 us; skew 500 ppm
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:53.037819 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:53.038194 4450 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:53.038268 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:53.038424 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:53.039499 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "a5626edc58d84a0bb5c5ca600146dbf9"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "83d3adc591ce4350fab69d3b3c5a0b7d"
server_key_iv: "19040c71dfc7cf934ecca44259502edb"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.039872 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5626edc58d84a0bb5c5ca600146dbf9"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "83d3adc591ce4350fab69d3b3c5a0b7d"
server_key_iv: "19040c71dfc7cf934ecca44259502edb"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.041235 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:53.042021 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.042285 4450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:53.042388 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "a5626edc58d84a0bb5c5ca600146dbf9"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "83d3adc591ce4350fab69d3b3c5a0b7d"
server_key_iv: "19040c71dfc7cf934ecca44259502edb"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.042470 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:53.063930 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:53.066434 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:53.066618 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:53.070784 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43391
I20260925 12:11:53.070827 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43391 every 8 connection(s)
I20260925 12:11:53.071147 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:53.072132 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:53.074605 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: Bootstrap starting.
I20260925 12:11:53.075294 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:53.075604 4517 log.cc:824] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:53.076273 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: No bootstrap required, opened a new log
I20260925 12:11:53.077880 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 12:11:53.078035 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:53.078058 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5626edc58d84a0bb5c5ca600146dbf9, State: Initialized, Role: FOLLOWER
I20260925 12:11:53.078210 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 12:11:53.078285 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:53.078308 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:53.078378 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:53.079217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4450
I20260925 12:11:53.079324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:53.079449 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 12:11:53.079538 392 external_mini_cluster.cc:1428] Setting key a9f987efbbe4697ad09cb71116702157
I20260925 12:11:53.079571 4517 leader_election.cc:302] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5626edc58d84a0bb5c5ca600146dbf9; no voters:
I20260925 12:11:53.079744 4517 leader_election.cc:288] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:53.079800 4520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:53.079967 4520 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [term 1 LEADER]: Becoming Leader. State: Replica: a5626edc58d84a0bb5c5ca600146dbf9, State: Running, Role: LEADER
I20260925 12:11:53.080045 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:53.080111 4520 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } }
I20260925 12:11:53.080542 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5626edc58d84a0bb5c5ca600146dbf9. Latest consensus state: current_term: 1 leader_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } } }
I20260925 12:11:53.080549 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5626edc58d84a0bb5c5ca600146dbf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43391 } } }
I20260925 12:11:53.080734 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:53.080844 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:53.081298 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:53.081868 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:53.083581 4531 catalog_manager.cc:1347] Generated new cluster ID: ce55716739534dc696844295af6f39ac
I20260925 12:11:53.083632 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:53.089140 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:53.089825 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:53.090344 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.080240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.0.98.62:43391)
0925 12:11:53.080451 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.080461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.080572 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:11:53.080808 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.080817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.080880 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.081022 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:53.081817 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.082942 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.084064 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.084555 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.084684 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.086958 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:53.086962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:53.086962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:53.086974 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:53.088797 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.089171 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.089174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.089175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.089298 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.089510 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.089511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.089511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.089732 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:53.089907 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.089977 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.090145 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
I20260925 12:11:53.092730 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: Generated new TSK 0
I20260925 12:11:53.092914 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:53.123672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:53.208138 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:53.208382 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:53.208423 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:53.209635 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:53.209682 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:53.209710 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:53.211225 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:53.211464 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:53.211750 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:53.216843 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.216843 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.216871 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:53.217082 4543 server_base.cc:1034] running on GCE node
I20260925 12:11:53.217309 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:53.217550 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:53.218690 4543 hybrid_clock.cc:648] HybridClock initialized: now 1790338313218680 us; error 28 us; skew 500 ppm
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:53.221390 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:53.221743 4543 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:53.221791 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:53.221880 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:53.222877 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "82d74fad763e4670a8ec025a6bde6ea3"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "9ed3696abf91eb0c774aad1709d2a366"
server_key_iv: "d679601c260c335e865d741d6e6b5b63"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.223229 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82d74fad763e4670a8ec025a6bde6ea3"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "9ed3696abf91eb0c774aad1709d2a366"
server_key_iv: "d679601c260c335e865d741d6e6b5b63"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.224436 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:53.225268 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.225488 4543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:53.225564 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "82d74fad763e4670a8ec025a6bde6ea3"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "9ed3696abf91eb0c774aad1709d2a366"
server_key_iv: "d679601c260c335e865d741d6e6b5b63"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.225660 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:53.245832 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:53.248289 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:53.248441 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:53.248740 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:53.249073 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:53.249128 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.249163 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:53.249189 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.255191 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37353
I20260925 12:11:53.255252 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37353 every 8 connection(s)
I20260925 12:11:53.255577 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:53.259289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4543
I20260925 12:11:53.259377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:53.259588 392 external_mini_cluster.cc:1428] Setting key b4f9434095bbc1265d60873d23f8894c
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:53.265728 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.256377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37051 (local address 127.0.98.62:43391)
0925 12:11:53.256427 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.256429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.256648 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:11:53.257230 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.257232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.257269 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.257325 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:53.258278 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.258801 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.259955 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.260179 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.260270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.262633 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:53.262634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:53.262634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:53.262642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:53.264123 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.264680 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.264681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.264681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.264733 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.265059 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.265060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.265060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.265173 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:53.265223 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.265590 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.265660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:11:53.266019 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.256330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43391 (local address 127.0.98.1:37051)
0925 12:11:53.256473 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.256542 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.257117 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.257396 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.257400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.257672 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:53.258169 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.258174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.258907 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.258908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.259865 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.259867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.260282 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.260283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.260407 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.261006 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:53.261015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:53.262522 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:53.264216 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.264224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.264227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.264601 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.264808 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.264809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.264809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.264977 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.265240 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:53.265246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:53.265441 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.265559 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.265738 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260925 12:11:53.266620 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43391
I20260925 12:11:53.266721 4671 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:53.266893 4671 heartbeater.cc:511] Master 127.0.98.62:43391 requested a full tablet report, sending...
I20260925 12:11:53.267463 4481 ts_manager.cc:194] Registered new tserver with Master: 82d74fad763e4670a8ec025a6bde6ea3 (127.0.98.1:37353)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:11:53.268181 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:53.288304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:53.375289 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:53.375494 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:53.375517 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:53.376744 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:53.376784 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:53.376811 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:53.378314 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:53.378557 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:53.378801 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:53.384228 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.384228 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.384258 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:53.384246 4676 server_base.cc:1034] running on GCE node
I20260925 12:11:53.384639 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:53.384853 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:53.385993 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790338313385982 us; error 31 us; skew 500 ppm
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:53.388779 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:53.389176 4676 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:53.389227 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:53.389320 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:53.390354 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "9efc014d90e840a2b7d1ef77ad6e6b10"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "7233a65dbc814d7f74296518b4641fc7"
server_key_iv: "1814f13b4527a41b48ba4e3dca77a206"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.390654 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9efc014d90e840a2b7d1ef77ad6e6b10"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "7233a65dbc814d7f74296518b4641fc7"
server_key_iv: "1814f13b4527a41b48ba4e3dca77a206"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.391903 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:11:53.392772 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.393034 4676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:53.393134 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "9efc014d90e840a2b7d1ef77ad6e6b10"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "7233a65dbc814d7f74296518b4641fc7"
server_key_iv: "1814f13b4527a41b48ba4e3dca77a206"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.393193 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:53.400303 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:53.402444 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:53.402565 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:53.402740 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:53.403111 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:53.403152 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.403175 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:53.403188 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.409829 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43485
I20260925 12:11:53.409862 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43485 every 8 connection(s)
I20260925 12:11:53.410154 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:53.413358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4676
I20260925 12:11:53.413472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:53.413661 392 external_mini_cluster.cc:1428] Setting key 58198c7796ab67555e034f329e4e35ed
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:53.420629 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.410901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50731 (local address 127.0.98.62:43391)
0925 12:11:53.410976 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.410978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.411218 (+ 240us) server_negotiation.cc:407] Connection header received
0925 12:11:53.411670 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.411671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.411681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.411739 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:53.412749 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.413260 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.414724 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.414925 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.415003 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.417518 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:53.417519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:53.417520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:53.417528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:53.419155 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.419622 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.419623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.419624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.419684 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.419947 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.419948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.419948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.420069 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:53.420154 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.420424 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.420529 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:11:53.420846 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.410889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43391 (local address 127.0.98.2:50731)
0925 12:11:53.411080 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.411116 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.411585 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.411793 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.411797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.412073 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:53.412655 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.412660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.413377 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.413378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.414623 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.414626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.415008 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.415009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.415142 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.415726 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:53.415735 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:53.417418 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:53.419290 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.419297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.419299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.419530 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.419767 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.419768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.419768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.419876 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.420144 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:53.420150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:53.420295 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.420406 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.420611 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260925 12:11:53.421355 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43391
I20260925 12:11:53.421437 4804 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:53.421597 4804 heartbeater.cc:511] Master 127.0.98.62:43391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:11:53.422009 4481 ts_manager.cc:194] Registered new tserver with Master: 9efc014d90e840a2b7d1ef77ad6e6b10 (127.0.98.2:43485)
I20260925 12:11:53.422416 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:53.440438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:53.537235 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:53.537452 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:53.537482 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:53.538756 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:53.538800 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:53.538829 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:53.540520 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:53.540831 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:53.541095 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:53.546818 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:53.546839 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:53.547143 4809 server_base.cc:1034] running on GCE node
W20260925 12:11:53.546834 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:53.547459 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:53.547672 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:53.548830 4809 hybrid_clock.cc:648] HybridClock initialized: now 1790338313548813 us; error 28 us; skew 500 ppm
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:53.551908 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:53.552328 4809 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:53.552381 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:53.552593 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:53.553779 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "91a61b4a047ec3fd242ffd13bf20c630"
server_key_iv: "30c9f95d549c77069b02f3f04509d3ca"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.554157 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "91a61b4a047ec3fd242ffd13bf20c630"
server_key_iv: "30c9f95d549c77069b02f3f04509d3ca"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.555631 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:53.556651 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.556866 4809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:53.557032 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d"
format_stamp: "Formatted at 2026-09-25 12:11:53 on dist-test-slave-2hqq"
server_key: "91a61b4a047ec3fd242ffd13bf20c630"
server_key_iv: "30c9f95d549c77069b02f3f04509d3ca"
server_key_version: "encryptionkey@0"
I20260925 12:11:53.557106 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:53.570886 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:53.573381 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:53.573545 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:53.573807 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:53.574148 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:53.574203 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.574245 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:53.574275 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:53.580470 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39395
I20260925 12:11:53.580570 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39395 every 8 connection(s)
I20260925 12:11:53.580892 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:53.586030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4809
I20260925 12:11:53.586153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:53.586398 392 external_mini_cluster.cc:1428] Setting key bb8c31602e54e9d70e05d739950aec1a
Sep 25 12:11:53 dist-test-slave-2hqq krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:53.591001 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.581705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45549 (local address 127.0.98.62:43391)
0925 12:11:53.581771 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.581774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.582094 (+ 320us) server_negotiation.cc:407] Connection header received
0925 12:11:53.582607 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.582608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.582618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.582672 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:53.583570 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.584074 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.585220 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.585405 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.585473 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.588216 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:53.588218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:53.588218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:53.588226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:53.589587 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.590011 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.590012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.590012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.590071 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.590339 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.590340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.590340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.590454 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:53.590553 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.590826 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.590905 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:11:53.591218 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.581699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43391 (local address 127.0.98.3:45549)
0925 12:11:53.581896 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.581997 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.582522 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.582728 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.582731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.582998 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:53.583494 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.583499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.584178 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.584179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.585119 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.585122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.585492 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.585492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.585599 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.586279 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:53.586288 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:53.588117 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:53.589686 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.589693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.589695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.589914 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.590146 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:53.590147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:53.590148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:53.590259 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:53.590543 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:53.590549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:53.590695 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.590806 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.590968 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 12:11:53.591737 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43391
I20260925 12:11:53.591835 4937 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:53.592008 4937 heartbeater.cc:511] Master 127.0.98.62:43391 requested a full tablet report, sending...
I20260925 12:11:53.592417 4481 ts_manager.cc:194] Registered new tserver with Master: e7ad2fbd33484d1fac7fef4d447f8f7d (127.0.98.3:39395)
I20260925 12:11:53.592855 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45549
I20260925 12:11:53.598866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:11:53.606833 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.600454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.62:43391)
0925 12:11:53.600523 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.600525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.600631 (+ 106us) server_negotiation.cc:407] Connection header received
0925 12:11:53.600675 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.600676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.600685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.600735 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:53.601321 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.601858 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.602575 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.602756 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.602815 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.604201 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:53.604203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:53.604203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:53.604209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:53.605649 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.605971 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.605972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.605972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.606035 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:53.606218 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:53.606218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:53.606219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:53.606427 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:53.606532 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.606601 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.606683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:11:53.608582 4481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:53.610227 4481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:53.617730 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.98.1:37353)
0925 12:11:53.613710 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.613714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.613724 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:11:53.613764 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.613766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.613783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.613893 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:53.614090 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615242 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.616677 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.617285 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.617360 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.617421 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.617606 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.617648 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":106,"threads_started":1}
I20260925 12:11:53.617733 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37353 (local address 127.0.0.1:51532)
0925 12:11:53.613447 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.613492 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.613573 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.613900 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.613904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.613906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:53.613999 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.614001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615369 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.615370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.616587 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.616590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.617383 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.617384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.617482 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.617499 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.617583 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.617639 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":173,"thread_start_us":60,"threads_started":1}
I20260925 12:11:53.618489 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39395 (local address 127.0.0.1:55576)
0925 12:11:53.613695 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.613705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.613763 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.614085 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.614086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.614088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:53.614151 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.614153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615385 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.615386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.617049 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.617051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.618251 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.618252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.618314 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.618318 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.618370 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.618409 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":164,"thread_start_us":84,"threads_started":1}
I20260925 12:11:53.618655 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43485 (local address 127.0.0.1:40384)
0925 12:11:53.613879 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.613888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.613980 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.614266 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.614267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.614269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:53.614306 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.614307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615339 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.615340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.617049 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.617051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.618399 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.618400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.618465 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.618470 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.618535 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.618579 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":189,"thread_start_us":83,"threads_started":1}
I20260925 12:11:53.618671 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.0.98.2:43485)
0925 12:11:53.613828 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.613833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.613941 (+ 108us) server_negotiation.cc:407] Connection header received
0925 12:11:53.614080 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.614082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.614098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.614153 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:53.614389 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615224 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.617170 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.618295 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.618406 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.618467 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.618549 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.618589 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260925 12:11:53.618768 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.613252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.98.3:39395)
0925 12:11:53.613444 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.613450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.613830 (+ 380us) server_negotiation.cc:407] Connection header received
0925 12:11:53.613879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.613882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.613914 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.613994 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:53.614233 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.615281 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.617143 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.617888 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.617965 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.618028 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.618642 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.618690 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 12:11:53.618687 4605 tablet_service.cc:1467] Processing CreateTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b (DEFAULT_TABLE table=test-table [id=aabdc43571cd431ebeda11d1282ed657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:53.619089 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40af8213bfc54f3c8ada875b2f0c7b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:53.619397 4738 tablet_service.cc:1467] Processing CreateTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b (DEFAULT_TABLE table=test-table [id=aabdc43571cd431ebeda11d1282ed657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:53.619407 4871 tablet_service.cc:1467] Processing CreateTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b (DEFAULT_TABLE table=test-table [id=aabdc43571cd431ebeda11d1282ed657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:53.619736 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40af8213bfc54f3c8ada875b2f0c7b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:53.619736 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40af8213bfc54f3c8ada875b2f0c7b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:53.623225 4955 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Bootstrap starting.
I20260925 12:11:53.623796 4955 tablet_bootstrap.cc:654] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:53.624111 4956 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Bootstrap starting.
I20260925 12:11:53.624253 4954 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Bootstrap starting.
I20260925 12:11:53.624444 4955 log.cc:824] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:53.624848 4956 tablet_bootstrap.cc:654] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:53.625006 4954 tablet_bootstrap.cc:654] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:53.625391 4955 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: No bootstrap required, opened a new log
I20260925 12:11:53.625483 4955 ts_tablet_manager.cc:1397] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:53.625525 4956 log.cc:824] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:53.625638 4954 log.cc:824] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:53.626514 4954 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: No bootstrap required, opened a new log
I20260925 12:11:53.626605 4954 ts_tablet_manager.cc:1397] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:53.627240 4955 raft_consensus.cc:348] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.627377 4955 raft_consensus.cc:374] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:53.627398 4955 raft_consensus.cc:729] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efc014d90e840a2b7d1ef77ad6e6b10, State: Initialized, Role: FOLLOWER
I20260925 12:11:53.627533 4955 consensus_queue.cc:260] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.627846 4955 ts_tablet_manager.cc:1428] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:11:53.627882 4804 heartbeater.cc:503] Master 127.0.98.62:43391 was elected leader, sending a full tablet report...
I20260925 12:11:53.628634 4956 tablet_bootstrap.cc:492] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: No bootstrap required, opened a new log
I20260925 12:11:53.628705 4956 ts_tablet_manager.cc:1397] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 12:11:53.628759 4954 raft_consensus.cc:348] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.628934 4954 raft_consensus.cc:374] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:53.628983 4954 raft_consensus.cc:729] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d74fad763e4670a8ec025a6bde6ea3, State: Initialized, Role: FOLLOWER
I20260925 12:11:53.629107 4954 consensus_queue.cc:260] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.629456 4954 ts_tablet_manager.cc:1428] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:11:53.629469 4671 heartbeater.cc:503] Master 127.0.98.62:43391 was elected leader, sending a full tablet report...
I20260925 12:11:53.630730 4956 raft_consensus.cc:348] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.630851 4956 raft_consensus.cc:374] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:53.630872 4956 raft_consensus.cc:729] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ad2fbd33484d1fac7fef4d447f8f7d, State: Initialized, Role: FOLLOWER
I20260925 12:11:53.631003 4956 consensus_queue.cc:260] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.631292 4956 ts_tablet_manager.cc:1428] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:11:53.631307 4937 heartbeater.cc:503] Master 127.0.98.62:43391 was elected leader, sending a full tablet report...
W20260925 12:11:53.660791 4805 tablet.cc:2369] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:53.701092 4960 raft_consensus.cc:482] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:53.701218 4960 raft_consensus.cc:504] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.701619 4960 leader_election.cc:288] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d74fad763e4670a8ec025a6bde6ea3 (127.0.98.1:37353), e7ad2fbd33484d1fac7fef4d447f8f7d (127.0.98.3:39395)
I20260925 12:11:53.705691 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.702095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48877 (local address 127.0.98.3:39395)
0925 12:11:53.702155 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.702157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.702164 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:11:53.702190 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.702191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.702200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.702250 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:53.702420 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.703120 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.704744 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.705385 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.705448 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.705464 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.705600 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.705634 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:11:53.705691 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.701985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47759 (local address 127.0.98.1:37353)
0925 12:11:53.702030 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.702032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.702043 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:53.702071 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.702071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.702082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.702139 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:11:53.702333 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.703112 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.704744 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.705303 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.705385 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.705396 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.705576 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.705621 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:11:53.705715 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.701798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37353 (local address 127.0.98.2:47759)
0925 12:11:53.701868 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.701880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.701996 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.702169 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.702170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.702173 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:53.702250 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.702252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.703205 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.703205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.704642 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.704645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.705391 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.705392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.705480 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.705485 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.705544 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.705591 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":74}
I20260925 12:11:53.705724 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.701805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39395 (local address 127.0.98.2:48877)
0925 12:11:53.702045 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.702065 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:53.702138 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:53.702276 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:53.702277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:53.702278 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:11:53.702344 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.702345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.703209 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.703209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.704642 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:53.704645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.705461 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.705462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:53.705525 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.705529 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:53.705580 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:11:53.705606 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":119,"thread_start_us":117,"threads_started":1}
I20260925 12:11:53.706159 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40af8213bfc54f3c8ada875b2f0c7b2b" candidate_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d74fad763e4670a8ec025a6bde6ea3" is_pre_election: true
I20260925 12:11:53.706153 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40af8213bfc54f3c8ada875b2f0c7b2b" candidate_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" is_pre_election: true
I20260925 12:11:53.706314 4891 raft_consensus.cc:2454] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efc014d90e840a2b7d1ef77ad6e6b10 in term 0.
I20260925 12:11:53.706564 4625 raft_consensus.cc:2454] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efc014d90e840a2b7d1ef77ad6e6b10 in term 0.
I20260925 12:11:53.706554 4693 leader_election.cc:302] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efc014d90e840a2b7d1ef77ad6e6b10, e7ad2fbd33484d1fac7fef4d447f8f7d; no voters:
I20260925 12:11:53.706722 4960 raft_consensus.cc:2781] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:53.706796 4960 raft_consensus.cc:482] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:53.706826 4960 raft_consensus.cc:3037] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 0 FOLLOWER]: Advancing to term 1
W20260925 12:11:53.756374 4672 tablet.cc:2369] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:53.781960 4960 raft_consensus.cc:504] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.782168 4960 leader_election.cc:288] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [CANDIDATE]: Term 1 election: Requested vote from peers 82d74fad763e4670a8ec025a6bde6ea3 (127.0.98.1:37353), e7ad2fbd33484d1fac7fef4d447f8f7d (127.0.98.3:39395)
I20260925 12:11:53.782418 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40af8213bfc54f3c8ada875b2f0c7b2b" candidate_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d"
I20260925 12:11:53.782421 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40af8213bfc54f3c8ada875b2f0c7b2b" candidate_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d74fad763e4670a8ec025a6bde6ea3"
I20260925 12:11:53.782532 4891 raft_consensus.cc:3037] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:53.782541 4625 raft_consensus.cc:3037] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:53.783294 4891 raft_consensus.cc:2454] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efc014d90e840a2b7d1ef77ad6e6b10 in term 1.
I20260925 12:11:53.783501 4693 leader_election.cc:302] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efc014d90e840a2b7d1ef77ad6e6b10, e7ad2fbd33484d1fac7fef4d447f8f7d; no voters:
I20260925 12:11:53.783634 4960 raft_consensus.cc:2781] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:53.783627 4625 raft_consensus.cc:2454] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efc014d90e840a2b7d1ef77ad6e6b10 in term 1.
I20260925 12:11:53.783831 4960 raft_consensus.cc:686] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 1 LEADER]: Becoming Leader. State: Replica: 9efc014d90e840a2b7d1ef77ad6e6b10, State: Running, Role: LEADER
I20260925 12:11:53.783938 4960 consensus_queue.cc:237] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } }
I20260925 12:11:53.785068 4481 catalog_manager.cc:5495] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efc014d90e840a2b7d1ef77ad6e6b10 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efc014d90e840a2b7d1ef77ad6e6b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43485 } health_report { overall_health: HEALTHY } } }
I20260925 12:11:53.804425 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.801910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.98.2:43485)
0925 12:11:53.802022 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:53.802027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:53.802035 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:53.802067 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:53.802068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:53.802088 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:53.802170 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:11:53.802313 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.802809 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.803613 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:53.803815 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:53.803890 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:53.804080 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:11:53.804181 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:11:53.804277 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:53.804296 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 12:11:53.804317 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:11:53.806258 4625 raft_consensus.cc:1261] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 1 FOLLOWER]: Refusing update from remote peer 9efc014d90e840a2b7d1ef77ad6e6b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:53.806259 4891 raft_consensus.cc:1261] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 1 FOLLOWER]: Refusing update from remote peer 9efc014d90e840a2b7d1ef77ad6e6b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:53.806663 4960 consensus_queue.cc:1035] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ad2fbd33484d1fac7fef4d447f8f7d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:53.806814 4964 consensus_queue.cc:1035] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d74fad763e4670a8ec025a6bde6ea3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:53.811028 4968 mvcc.cc:205] Tried to move back new op lower bound from 7333225733347729408 to 7333225733260427264. Current Snapshot: MvccSnapshot[applied={T|T < 7333225733347729408 or (T in {7333225733347729408})}]
I20260925 12:11:53.811228 4481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:53.811326 4481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:53.812055 4481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5626edc58d84a0bb5c5ca600146dbf9: Sending DeleteTablet for 3 replicas of tablet 40af8213bfc54f3c8ada875b2f0c7b2b
I20260925 12:11:53.812359 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51532
I20260925 12:11:53.812402 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55576
I20260925 12:11:53.812623 4976 tablet_replica.cc:330] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3: stopping tablet replica
I20260925 12:11:53.812719 4976 raft_consensus.cc:2229] T 40af8213bfc54f3c8ada875b2f0c7b2b P 82d74fad763e4670a8ec025a6bde6ea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:53.812742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4543
I20260925 12:11:53.815505 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet 40af8213bfc54f3c8ada875b2f0c7b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40384
I20260925 12:11:53.815755 4978 tablet_replica.cc:330] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: stopping tablet replica
I20260925 12:11:53.815868 4978 raft_consensus.cc:2229] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:11:53.815970 4978 raft_consensus.cc:2258] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:53.816352 4978 ts_tablet_manager.cc:1905] T 40af8213bfc54f3c8ada875b2f0c7b2b P 9efc014d90e840a2b7d1ef77ad6e6b10: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 12:11:53.819408 4468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37353 (error 108)
I20260925 12:11:53.819535 4977 tablet_replica.cc:330] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: stopping tablet replica
I20260925 12:11:53.819639 4977 raft_consensus.cc:2229] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:53.819722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4676
I20260925 12:11:53.819772 4977 raft_consensus.cc:2258] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:53.821041 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.820611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37353 (local address 127.0.0.1:51548)
0925 12:11:53.820973 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.821011 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
W20260925 12:11:53.821174 4468 catalog_manager.cc:4584] TS 82d74fad763e4670a8ec025a6bde6ea3 (127.0.98.1:37353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40af8213bfc54f3c8ada875b2f0c7b2b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37353: connect: Connection refused (error 111)
I20260925 12:11:53.821781 4977 ts_tablet_manager.cc:1905] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:53.874042 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.873876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37353 (local address 127.0.0.1:51554)
0925 12:11:53.873938 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.873985 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 12:11:53.874455 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.874376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37353 (local address 127.0.0.1:51558)
0925 12:11:53.874411 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.874432 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 12:11:53.874810 4977 ts_tablet_manager.cc:1918] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:53.874917 4977 log.cc:1196] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/40af8213bfc54f3c8ada875b2f0c7b2b
I20260925 12:11:53.875322 4977 ts_tablet_manager.cc:1939] T 40af8213bfc54f3c8ada875b2f0c7b2b P e7ad2fbd33484d1fac7fef4d447f8f7d: Deleting consensus metadata
I20260925 12:11:53.877336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4809
I20260925 12:11:53.877482 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:53.877295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43485 (local address 127.0.0.1:40400)
0925 12:11:53.877373 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:53.877408 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 12:11:53.891031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4450
2026-09-25T12:11:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2104 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T12:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:53Z Disabled control of system clock
I20260925 12:11:53.920732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37627
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:11:54.014853 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.015100 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.015126 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.016387 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:54.016430 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.016444 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.016458 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:54.016472 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:54.018033 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.018262 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.018539 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:54.024204 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.024223 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.024257 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.024560 4983 server_base.cc:1034] running on GCE node
I20260925 12:11:54.024825 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.025131 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.026290 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790338314026264 us; error 37 us; skew 500 ppm
I20260925 12:11:54.027601 4983 webserver.cc:460] Webserver started at http://127.0.98.62:44469/ using document root <none> and password file <none>
I20260925 12:11:54.027944 4983 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.028003 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.028159 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.029412 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "0a797ba913e246eab2e7fd0ec439a40d"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6f2e6087b6e5d49b7f6bd9c93b6b9321"
server_key_iv: "5d99e3f140c9f0ef835da1d33987fe29"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.029807 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a797ba913e246eab2e7fd0ec439a40d"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6f2e6087b6e5d49b7f6bd9c93b6b9321"
server_key_iv: "5d99e3f140c9f0ef835da1d33987fe29"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.031278 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:11:54.032051 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.032258 4983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:54.032347 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "0a797ba913e246eab2e7fd0ec439a40d"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6f2e6087b6e5d49b7f6bd9c93b6b9321"
server_key_iv: "5d99e3f140c9f0ef835da1d33987fe29"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.032400 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.035674 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.035903 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.036001 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.039615 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37627
I20260925 12:11:54.039675 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37627 every 8 connection(s)
I20260925 12:11:54.040019 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:54.040755 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.043471 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: Bootstrap starting.
I20260925 12:11:54.044066 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:54.044365 5050 log.cc:824] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:54.045100 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: No bootstrap required, opened a new log
I20260925 12:11:54.045733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 4983
I20260925 12:11:54.045826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:54.046010 392 external_mini_cluster.cc:1428] Setting key 45044aad9ccffeb15541f3e31141b90b
I20260925 12:11:54.047520 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260925 12:11:54.047698 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:54.047734 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a797ba913e246eab2e7fd0ec439a40d, State: Initialized, Role: FOLLOWER
I20260925 12:11:54.047891 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260925 12:11:54.047984 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:54.048017 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:54.048058 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:54.048918 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.046546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.0.98.62:37627)
0925 12:11:54.046782 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.046791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.046826 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:11:54.047465 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.047474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.047525 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.047676 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.048455 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.048458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.048459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.048463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.048535 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.048548 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.048635 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.048706 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.048764 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260925 12:11:54.049098 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260925 12:11:54.049202 5050 leader_election.cc:302] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a797ba913e246eab2e7fd0ec439a40d; no voters:
I20260925 12:11:54.049388 5050 leader_election.cc:288] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:54.049417 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:54.049531 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [term 1 LEADER]: Becoming Leader. State: Replica: 0a797ba913e246eab2e7fd0ec439a40d, State: Running, Role: LEADER
I20260925 12:11:54.049610 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260925 12:11:54.049666 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:54.049943 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a797ba913e246eab2e7fd0ec439a40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } } }
I20260925 12:11:54.050012 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:54.050132 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a797ba913e246eab2e7fd0ec439a40d. Latest consensus state: current_term: 1 leader_uuid: "0a797ba913e246eab2e7fd0ec439a40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a797ba913e246eab2e7fd0ec439a40d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } } }
I20260925 12:11:54.050196 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d [sys.catalog]: This master's current role is: LEADER
W20260925 12:11:54.051368 5071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:11:54.051432 5071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:11:54.051491 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:54.051668 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:54.053232 5060 catalog_manager.cc:1347] Generated new cluster ID: 51f9db42f4cb43faa88860a5dd49d337
I20260925 12:11:54.053296 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:54.081564 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:54.081784 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:54.087211 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: Generated new TSK 0
I20260925 12:11:54.087442 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:11:54.093125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--builtin_ntp_servers=127.0.98.20:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:11:54.173151 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.173350 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.173372 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.174532 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.174572 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.174598 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:54.175931 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.176187 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.176503 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:54.181515 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.181578 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.181643 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.181736 5074 server_base.cc:1034] running on GCE node
I20260925 12:11:54.182088 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.182355 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.183513 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790338314183495 us; error 42 us; skew 500 ppm
I20260925 12:11:54.184907 5074 webserver.cc:460] Webserver started at http://127.0.98.1:44939/ using document root <none> and password file <none>
I20260925 12:11:54.185173 5074 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.185216 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.185343 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.186399 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "6573a0706f8444a9aaa9c0f7915c8ea1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "c29eea1fc47efa447cf72f53ba23c898"
server_key_iv: "28a956e3aa2f4287da7658c93546b2b4"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.186753 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6573a0706f8444a9aaa9c0f7915c8ea1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "c29eea1fc47efa447cf72f53ba23c898"
server_key_iv: "28a956e3aa2f4287da7658c93546b2b4"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.188117 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:54.188879 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.189045 5074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.189141 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "6573a0706f8444a9aaa9c0f7915c8ea1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "c29eea1fc47efa447cf72f53ba23c898"
server_key_iv: "28a956e3aa2f4287da7658c93546b2b4"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.189218 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.192565 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.192873 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.193022 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.193231 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:54.193526 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:54.193578 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.193616 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:54.193643 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.199515 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34933
I20260925 12:11:54.199551 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34933 every 8 connection(s)
I20260925 12:11:54.199898 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:54.202817 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.200677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41659 (local address 127.0.98.62:37627)
0925 12:11:54.200767 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.200769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.201009 (+ 240us) server_negotiation.cc:407] Connection header received
0925 12:11:54.201422 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.201424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.201428 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.201487 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.202618 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.202619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.202620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.202623 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.202636 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.202640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.202689 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.202752 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.202765 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 12:11:54.203014 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.200667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37627 (local address 127.0.98.1:41659)
0925 12:11:54.200873 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.200906 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.201339 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.201538 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.201541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.201548 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.202471 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.202472 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.202499 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.202528 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.202686 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.202689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.202691 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.202741 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.202786 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260925 12:11:54.203558 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37627
I20260925 12:11:54.203631 5202 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:54.203789 5202 heartbeater.cc:511] Master 127.0.98.62:37627 requested a full tablet report, sending...
I20260925 12:11:54.204284 5014 ts_manager.cc:194] Registered new tserver with Master: 6573a0706f8444a9aaa9c0f7915c8ea1 (127.0.98.1:34933)
I20260925 12:11:54.207966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5074
I20260925 12:11:54.208057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:54.208274 392 external_mini_cluster.cc:1428] Setting key e8b4c035ee54d06e56dd05799009e2b2
I20260925 12:11:54.209201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--builtin_ntp_servers=127.0.98.20:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 12:11:54.293814 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.294158 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.294193 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.295431 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.295472 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.295499 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:54.296908 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.297202 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.297525 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:54.303181 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.303192 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.303256 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.303442 5205 server_base.cc:1034] running on GCE node
I20260925 12:11:54.303737 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.303947 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.305141 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790338314305128 us; error 25 us; skew 500 ppm
I20260925 12:11:54.306360 5205 webserver.cc:460] Webserver started at http://127.0.98.2:33375/ using document root <none> and password file <none>
I20260925 12:11:54.306631 5205 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.306703 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.306857 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.307926 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "272c236dc7614e03b305c0354201288e"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "70b51974cf624d9076b99538737e8b89"
server_key_iv: "43f84fb3a67b0af2bad3c5ea29fd88de"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.308287 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "272c236dc7614e03b305c0354201288e"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "70b51974cf624d9076b99538737e8b89"
server_key_iv: "43f84fb3a67b0af2bad3c5ea29fd88de"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.309682 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:54.310497 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.310667 5205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:54.310765 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "272c236dc7614e03b305c0354201288e"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "70b51974cf624d9076b99538737e8b89"
server_key_iv: "43f84fb3a67b0af2bad3c5ea29fd88de"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.310846 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.314278 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.314558 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.314699 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.314908 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:54.315207 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:54.315258 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.315296 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:54.315323 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.321018 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35243
I20260925 12:11:54.321070 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35243 every 8 connection(s)
I20260925 12:11:54.321421 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:54.323835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5205
I20260925 12:11:54.323951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:54.324213 392 external_mini_cluster.cc:1428] Setting key 5a9f335ee54867ba5c93bf125954a1a3
I20260925 12:11:54.324782 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.322174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42089 (local address 127.0.98.62:37627)
0925 12:11:54.322247 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.322249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.322480 (+ 231us) server_negotiation.cc:407] Connection header received
0925 12:11:54.322918 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.322919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.322924 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.322985 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.324392 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.324393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.324394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.324399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.324415 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.324419 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.324581 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.324699 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.324714 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 12:11:54.325003 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.322152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37627 (local address 127.0.98.2:42089)
0925 12:11:54.322347 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.322387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.322835 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.323060 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.323063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.323071 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.324164 (+ 1093us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.324165 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.324205 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.324243 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.324615 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.324621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.324626 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.324713 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.324761 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260925 12:11:54.325275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--builtin_ntp_servers=127.0.98.20:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 12:11:54.325438 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37627
I20260925 12:11:54.325519 5333 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:54.325663 5333 heartbeater.cc:511] Master 127.0.98.62:37627 requested a full tablet report, sending...
I20260925 12:11:54.326013 5014 ts_manager.cc:194] Registered new tserver with Master: 272c236dc7614e03b305c0354201288e (127.0.98.2:35243)
W20260925 12:11:54.406044 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.406257 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.406281 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.407491 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.407536 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.407564 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:54.409016 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.409278 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.409585 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:54.414876 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.414863 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.414878 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.415108 5336 server_base.cc:1034] running on GCE node
I20260925 12:11:54.415316 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.415522 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.416657 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790338314416638 us; error 33 us; skew 500 ppm
I20260925 12:11:54.417994 5336 webserver.cc:460] Webserver started at http://127.0.98.3:37015/ using document root <none> and password file <none>
I20260925 12:11:54.418234 5336 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.418280 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.418452 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.419545 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "e8bbbfb9940143bfaa57a98f11c18db1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "94540140993f3b74963f48917bf94e6e"
server_key_iv: "4af5099ae35837b77a736b66bc67a289"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.419911 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e8bbbfb9940143bfaa57a98f11c18db1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "94540140993f3b74963f48917bf94e6e"
server_key_iv: "4af5099ae35837b77a736b66bc67a289"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.421350 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:54.422096 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.422264 5336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:54.422367 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "e8bbbfb9940143bfaa57a98f11c18db1"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "94540140993f3b74963f48917bf94e6e"
server_key_iv: "4af5099ae35837b77a736b66bc67a289"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.422443 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.425845 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.426117 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.426260 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.426481 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:54.426777 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:54.426831 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.426869 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:54.426895 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.433059 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33925
I20260925 12:11:54.433450 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:54.436524 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33925 every 8 connection(s)
I20260925 12:11:54.439720 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.437665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55029 (local address 127.0.98.62:37627)
0925 12:11:54.437725 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.437727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.437738 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:54.438220 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.438221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.438227 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.438280 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.439490 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.439491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.439491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.439495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.439510 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.439514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.439561 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.439634 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.439650 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:11:54.439918 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.437305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37627 (local address 127.0.98.3:55029)
0925 12:11:54.437533 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.437576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.438101 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.438351 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.438355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.438365 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.439328 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.439329 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.439356 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.439386 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.439558 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.439562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.439566 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.439644 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.439703 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":125,"threads_started":1}
I20260925 12:11:54.440179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5336
I20260925 12:11:54.440292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:54.440394 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37627
I20260925 12:11:54.440507 5464 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:54.440539 392 external_mini_cluster.cc:1428] Setting key be7e2b6ab315115ebc1562bb51d36444
I20260925 12:11:54.440662 5464 heartbeater.cc:511] Master 127.0.98.62:37627 requested a full tablet report, sending...
I20260925 12:11:54.441064 5014 ts_manager.cc:194] Registered new tserver with Master: e8bbbfb9940143bfaa57a98f11c18db1 (127.0.98.3:33925)
I20260925 12:11:54.441462 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:11:54.443804 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.442796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.0.98.62:37627)
0925 12:11:54.442892 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.442893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.442951 (+ 58us) server_negotiation.cc:407] Connection header received
0925 12:11:54.443001 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.443001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.443005 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.443063 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.443654 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.443655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.443655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.443658 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.443669 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.443672 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.443717 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.443748 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.443759 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:11:54.445842 5013 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:11:54.447489 5013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:11:54.451541 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.449835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.0.98.3:33925)
0925 12:11:54.450075 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.450080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.450354 (+ 274us) server_negotiation.cc:407] Connection header received
0925 12:11:54.450382 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.450385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.450394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.450466 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.451203 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.451211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.451211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.451216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.451253 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.451278 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.451351 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.451447 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.451462 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260925 12:11:54.451530 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.449776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35243 (local address 127.0.0.1:33058)
0925 12:11:54.450209 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.450220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.450283 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.450774 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.450775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.450778 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.451070 (+ 292us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.451071 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.451084 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.451107 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.451380 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.451381 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.451382 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.451438 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.451452 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":182,"threads_started":1}
I20260925 12:11:54.451886 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.449856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.0.98.2:35243)
0925 12:11:54.450146 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.450166 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.450353 (+ 187us) server_negotiation.cc:407] Connection header received
0925 12:11:54.450391 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.450393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.450402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.450470 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.451203 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.451211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.451211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.451216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.451253 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.451278 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.451351 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.451722 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.451736 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260925 12:11:54.452049 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.449771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33925 (local address 127.0.0.1:43868)
0925 12:11:54.450031 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.450221 (+ 190us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.450283 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.450526 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.450531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.450537 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.451068 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.451069 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.451084 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.451107 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.451357 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.451363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.451365 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.451439 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.451458 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":6,"thread_start_us":54,"threads_started":1}
I20260925 12:11:54.452708 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 2c42177ae225436aa76c20606990405a (DEFAULT_TABLE table=test-table [id=b946e2f6d7fc4a1fb4332f883f00337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:54.452814 5398 tablet_service.cc:1467] Processing CreateTablet for tablet 2c42177ae225436aa76c20606990405a (DEFAULT_TABLE table=test-table [id=b946e2f6d7fc4a1fb4332f883f00337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:54.453097 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c42177ae225436aa76c20606990405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.453148 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c42177ae225436aa76c20606990405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.455684 5478 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Bootstrap starting.
I20260925 12:11:54.455706 5479 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Bootstrap starting.
I20260925 12:11:54.456151 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.450089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34933 (local address 127.0.0.1:42930)
0925 12:11:54.450397 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.450409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.450455 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.451649 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.451650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.451653 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.452068 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.452069 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.452075 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.452087 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.455999 (+ 3912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.456000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.456001 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.456067 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.456084 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":63,"threads_started":1}
I20260925 12:11:54.456182 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.450282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.0.98.1:34933)
0925 12:11:54.451258 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.451262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.451270 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:11:54.451291 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.451292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.451299 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.451385 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.452179 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.452187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.452187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.452192 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.452231 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.455929 (+ 3698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.456007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.456076 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.456096 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":117,"threads_started":1}
I20260925 12:11:54.456424 5479 tablet_bootstrap.cc:654] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:54.456640 5478 tablet_bootstrap.cc:654] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:54.456804 5479 log.cc:824] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:54.456941 5478 log.cc:824] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:54.456988 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 2c42177ae225436aa76c20606990405a (DEFAULT_TABLE table=test-table [id=b946e2f6d7fc4a1fb4332f883f00337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:11:54.457329 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c42177ae225436aa76c20606990405a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.457764 5479 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: No bootstrap required, opened a new log
I20260925 12:11:54.457764 5478 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: No bootstrap required, opened a new log
I20260925 12:11:54.457832 5478 ts_tablet_manager.cc:1397] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 12:11:54.457932 5479 ts_tablet_manager.cc:1397] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:11:54.459869 5482 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: Bootstrap starting.
I20260925 12:11:54.460426 5482 tablet_bootstrap.cc:654] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:54.460553 5479 raft_consensus.cc:348] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.460553 5478 raft_consensus.cc:348] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.460723 5479 raft_consensus.cc:374] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:54.460723 5478 raft_consensus.cc:374] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:54.460765 5478 raft_consensus.cc:729] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272c236dc7614e03b305c0354201288e, State: Initialized, Role: FOLLOWER
I20260925 12:11:54.460765 5479 raft_consensus.cc:729] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bbbfb9940143bfaa57a98f11c18db1, State: Initialized, Role: FOLLOWER
I20260925 12:11:54.460772 5482 log.cc:824] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:54.460901 5478 consensus_queue.cc:260] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.460901 5479 consensus_queue.cc:260] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.461167 5479 ts_tablet_manager.cc:1428] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:11:54.461167 5478 ts_tablet_manager.cc:1428] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260925 12:11:54.461289 5333 heartbeater.cc:503] Master 127.0.98.62:37627 was elected leader, sending a full tablet report...
I20260925 12:11:54.461297 5464 heartbeater.cc:503] Master 127.0.98.62:37627 was elected leader, sending a full tablet report...
I20260925 12:11:54.461928 5482 tablet_bootstrap.cc:492] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: No bootstrap required, opened a new log
I20260925 12:11:54.462000 5482 ts_tablet_manager.cc:1397] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:11:54.463740 5482 raft_consensus.cc:348] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.463898 5482 raft_consensus.cc:374] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:54.463943 5482 raft_consensus.cc:729] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6573a0706f8444a9aaa9c0f7915c8ea1, State: Initialized, Role: FOLLOWER
I20260925 12:11:54.464057 5482 consensus_queue.cc:260] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.464296 5482 ts_tablet_manager.cc:1428] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:11:54.464359 5202 heartbeater.cc:503] Master 127.0.98.62:37627 was elected leader, sending a full tablet report...
I20260925 12:11:54.542655 5486 raft_consensus.cc:482] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:11:54.542819 5486 raft_consensus.cc:504] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.543195 5486 leader_election.cc:288] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 272c236dc7614e03b305c0354201288e (127.0.98.2:35243), e8bbbfb9940143bfaa57a98f11c18db1 (127.0.98.3:33925)
I20260925 12:11:54.544574 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.543413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52745 (local address 127.0.98.2:35243)
0925 12:11:54.543489 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.543576 (+ 87us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.543587 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:11:54.543618 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.543619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.543625 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.543688 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.544273 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.544274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.544275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.544279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.544301 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.544306 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.544413 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.544472 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.544510 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 12:11:54.544571 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.543356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35243 (local address 127.0.98.1:52745)
0925 12:11:54.543418 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.543431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.543469 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.543707 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.543708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.543711 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.544146 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.544146 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.544152 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.544163 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.544416 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.544417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.544418 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.544469 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.544503 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 12:11:54.544693 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.543545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51005 (local address 127.0.98.3:33925)
0925 12:11:54.543614 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.543616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.543738 (+ 122us) server_negotiation.cc:407] Connection header received
0925 12:11:54.543765 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.543766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.543772 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.543819 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.544367 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.544369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.544369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.544373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.544405 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.544409 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.544483 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.544604 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.544634 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:11:54.544696 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.543426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33925 (local address 127.0.98.1:51005)
0925 12:11:54.543634 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.543643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.543715 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.543843 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.543844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.543846 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:54.544275 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:11:54.544276 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:54.544280 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:11:54.544289 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:54.544481 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:54.544482 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:54.544483 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:11:54.544545 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:11:54.544559 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260925 12:11:54.545001 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c42177ae225436aa76c20606990405a" candidate_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c236dc7614e03b305c0354201288e" is_pre_election: true
I20260925 12:11:54.545152 5287 raft_consensus.cc:2454] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6573a0706f8444a9aaa9c0f7915c8ea1 in term 0.
I20260925 12:11:54.545409 5092 leader_election.cc:302] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c236dc7614e03b305c0354201288e, 6573a0706f8444a9aaa9c0f7915c8ea1; no voters:
I20260925 12:11:54.545531 5486 raft_consensus.cc:2781] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:11:54.545511 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c42177ae225436aa76c20606990405a" candidate_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" is_pre_election: true
I20260925 12:11:54.545586 5486 raft_consensus.cc:482] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:11:54.545634 5486 raft_consensus.cc:3037] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:54.545645 5418 raft_consensus.cc:2454] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6573a0706f8444a9aaa9c0f7915c8ea1 in term 0.
I20260925 12:11:54.546351 5486 raft_consensus.cc:504] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.546490 5486 leader_election.cc:288] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [CANDIDATE]: Term 1 election: Requested vote from peers 272c236dc7614e03b305c0354201288e (127.0.98.2:35243), e8bbbfb9940143bfaa57a98f11c18db1 (127.0.98.3:33925)
I20260925 12:11:54.546610 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c42177ae225436aa76c20606990405a" candidate_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272c236dc7614e03b305c0354201288e"
I20260925 12:11:54.546634 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c42177ae225436aa76c20606990405a" candidate_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bbbfb9940143bfaa57a98f11c18db1"
I20260925 12:11:54.546690 5287 raft_consensus.cc:3037] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:54.546737 5418 raft_consensus.cc:3037] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:54.547317 5287 raft_consensus.cc:2454] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6573a0706f8444a9aaa9c0f7915c8ea1 in term 1.
I20260925 12:11:54.547475 5092 leader_election.cc:302] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 272c236dc7614e03b305c0354201288e, 6573a0706f8444a9aaa9c0f7915c8ea1; no voters:
I20260925 12:11:54.547554 5486 raft_consensus.cc:2781] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:54.547552 5418 raft_consensus.cc:2454] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6573a0706f8444a9aaa9c0f7915c8ea1 in term 1.
I20260925 12:11:54.547694 5486 raft_consensus.cc:686] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 6573a0706f8444a9aaa9c0f7915c8ea1, State: Running, Role: LEADER
I20260925 12:11:54.547789 5486 consensus_queue.cc:237] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } }
I20260925 12:11:54.548661 5013 catalog_manager.cc:5495] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6573a0706f8444a9aaa9c0f7915c8ea1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6573a0706f8444a9aaa9c0f7915c8ea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 } health_report { overall_health: UNKNOWN } } }
I20260925 12:11:54.568707 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.567391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.1:34933)
0925 12:11:54.567448 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.567449 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.567479 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:11:54.567564 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.567566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.567579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.567630 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.568344 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.568345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.568345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.568349 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.568364 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.568369 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.568425 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.568587 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.568604 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:11:54.570571 5287 raft_consensus.cc:1261] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 1 FOLLOWER]: Refusing update from remote peer 6573a0706f8444a9aaa9c0f7915c8ea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:54.570592 5418 raft_consensus.cc:1261] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 1 FOLLOWER]: Refusing update from remote peer 6573a0706f8444a9aaa9c0f7915c8ea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:11:54.570943 5486 consensus_queue.cc:1035] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bbbfb9940143bfaa57a98f11c18db1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:11:54.571081 5488 consensus_queue.cc:1035] T 2c42177ae225436aa76c20606990405a P 6573a0706f8444a9aaa9c0f7915c8ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272c236dc7614e03b305c0354201288e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:11:54.571902 5334 tablet.cc:2369] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:11:54.573598 5491 mvcc.cc:205] Tried to move back new op lower bound from 7333225736478416896 to 7333225736388546560. Current Snapshot: MvccSnapshot[applied={T|T < 7333225736478416896}]
I20260925 12:11:54.573668 5492 mvcc.cc:205] Tried to move back new op lower bound from 7333225736478416896 to 7333225736388546560. Current Snapshot: MvccSnapshot[applied={T|T < 7333225736478416896}]
I20260925 12:11:54.574038 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333225736478416896 to 7333225736388546560. Current Snapshot: MvccSnapshot[applied={T|T < 7333225736478416896}]
I20260925 12:11:54.576226 5013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:54.576334 5013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:11:54.577126 5013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a797ba913e246eab2e7fd0ec439a40d: Sending DeleteTablet for 3 replicas of tablet 2c42177ae225436aa76c20606990405a
I20260925 12:11:54.577481 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c42177ae225436aa76c20606990405a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:54 UTC) from {username='slave'} at 127.0.0.1:33058
I20260925 12:11:54.577492 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c42177ae225436aa76c20606990405a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:54 UTC) from {username='slave'} at 127.0.0.1:43868
I20260925 12:11:54.577708 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c42177ae225436aa76c20606990405a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:11:54 UTC) from {username='slave'} at 127.0.0.1:42930
I20260925 12:11:54.577725 5501 tablet_replica.cc:330] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: stopping tablet replica
I20260925 12:11:54.577829 5501 raft_consensus.cc:2229] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:54.577939 5501 raft_consensus.cc:2258] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:54.578091 5500 tablet_replica.cc:330] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: stopping tablet replica
I20260925 12:11:54.578094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5074
I20260925 12:11:54.578191 5500 raft_consensus.cc:2229] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:11:54.578871 5500 raft_consensus.cc:2258] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:11:54.579365 5500 ts_tablet_manager.cc:1905] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:54.579355 5501 ts_tablet_manager.cc:1905] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:11:54.581379 5500 ts_tablet_manager.cc:1918] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:54.581399 5501 ts_tablet_manager.cc:1918] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:11:54.581457 5501 log.cc:1196] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/2c42177ae225436aa76c20606990405a
I20260925 12:11:54.581458 5500 log.cc:1196] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/2c42177ae225436aa76c20606990405a
I20260925 12:11:54.581730 5500 ts_tablet_manager.cc:1939] T 2c42177ae225436aa76c20606990405a P 272c236dc7614e03b305c0354201288e: Deleting consensus metadata
I20260925 12:11:54.581730 5501 ts_tablet_manager.cc:1939] T 2c42177ae225436aa76c20606990405a P e8bbbfb9940143bfaa57a98f11c18db1: Deleting consensus metadata
I20260925 12:11:54.582139 5001 catalog_manager.cc:4841] TS 272c236dc7614e03b305c0354201288e (127.0.98.2:35243): tablet 2c42177ae225436aa76c20606990405a (table test-table [id=b946e2f6d7fc4a1fb4332f883f00337d]) successfully deleted
I20260925 12:11:54.582203 4999 catalog_manager.cc:4841] TS e8bbbfb9940143bfaa57a98f11c18db1 (127.0.98.3:33925): tablet 2c42177ae225436aa76c20606990405a (table test-table [id=b946e2f6d7fc4a1fb4332f883f00337d]) successfully deleted
W20260925 12:11:54.583755 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34933 (error 108)
I20260925 12:11:54.584241 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.584149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34933 (local address 127.0.0.1:42946)
0925 12:11:54.584182 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.584216 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 12:11:54.584283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5205
W20260925 12:11:54.584398 5001 catalog_manager.cc:4584] TS 6573a0706f8444a9aaa9c0f7915c8ea1 (127.0.98.1:34933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c42177ae225436aa76c20606990405a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34933: connect: Connection refused (error 111)
I20260925 12:11:54.589540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5336
I20260925 12:11:54.593974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 4983
2026-09-25T12:11:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (700 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 12:11:54.613315 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:11:54.614375 392 webserver.cc:460] Webserver started at https://127.0.0.1:43301/ using document root <none> and password file <none>
I20260925 12:11:54.614504 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:11:54.614688 392 webserver.cc:460] Webserver started at http://127.0.0.1:36919/ using document root <none> and password file <none>
2026-09-25T12:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:54Z Disabled control of system clock
I20260925 12:11:54.622778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43301/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/testchainca.cert with env {}
W20260925 12:11:54.704392 5509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.704644 5509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.704668 5509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.705828 5509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:54.705878 5509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.705890 5509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.705904 5509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:54.705915 5509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:11:54.705935 5509 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:11:54.705946 5509 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43301/jwks.json
I20260925 12:11:54.707417 5509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43301/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.707615 5509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.707794 5509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:11:54.712988 5509 server_base.cc:1034] running on GCE node
W20260925 12:11:54.713017 5514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.712926 5515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.712930 5517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.713413 5509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.713680 5509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.714828 5509 hybrid_clock.cc:648] HybridClock initialized: now 1790338314714813 us; error 29 us; skew 500 ppm
I20260925 12:11:54.716142 5509 webserver.cc:460] Webserver started at http://127.0.98.62:42155/ using document root <none> and password file <none>
I20260925 12:11:54.716414 5509 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.716465 5509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.716641 5509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.717728 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "777a27920ebc41e0be00418264afcfb7"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "7789ba47df4286dad20223c1d0ada329"
server_key_iv: "31013ce507468defb9d13b88ead78f8e"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.718097 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "777a27920ebc41e0be00418264afcfb7"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "7789ba47df4286dad20223c1d0ada329"
server_key_iv: "31013ce507468defb9d13b88ead78f8e"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.719523 5509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:54.720269 5523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.720455 5509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:54.720578 5509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "777a27920ebc41e0be00418264afcfb7"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "7789ba47df4286dad20223c1d0ada329"
server_key_iv: "31013ce507468defb9d13b88ead78f8e"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.720665 5509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.736404 5509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.736791 5509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.736953 5509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.741278 5509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44095
I20260925 12:11:54.741313 5575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44095 every 8 connection(s)
I20260925 12:11:54.741711 5509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:54.742688 5576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.745044 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7: Bootstrap starting.
I20260925 12:11:54.745648 5576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:54.745942 5576 log.cc:824] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:54.746632 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7: No bootstrap required, opened a new log
I20260925 12:11:54.747255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5509
I20260925 12:11:54.747362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:54.747555 392 external_mini_cluster.cc:1428] Setting key 5da3906df568acf0f82809ebfa878903
I20260925 12:11:54.749014 5576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260925 12:11:54.749215 5576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:54.749259 5576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777a27920ebc41e0be00418264afcfb7, State: Initialized, Role: FOLLOWER
I20260925 12:11:54.749435 5576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260925 12:11:54.749527 5576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:54.749552 5576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:54.749606 5576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:54.750191 5576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260925 12:11:54.750275 5576 leader_election.cc:302] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 777a27920ebc41e0be00418264afcfb7; no voters:
I20260925 12:11:54.750466 5576 leader_election.cc:288] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:54.750520 5581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:54.750651 5581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [term 1 LEADER]: Becoming Leader. State: Replica: 777a27920ebc41e0be00418264afcfb7, State: Running, Role: LEADER
I20260925 12:11:54.750742 5576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:54.750756 5581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260925 12:11:54.751161 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 777a27920ebc41e0be00418264afcfb7. Latest consensus state: current_term: 1 leader_uuid: "777a27920ebc41e0be00418264afcfb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260925 12:11:54.751241 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:54.751346 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "777a27920ebc41e0be00418264afcfb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777a27920ebc41e0be00418264afcfb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260925 12:11:54.751442 5585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:54.751499 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:54.751957 5585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:54.754276 5585 catalog_manager.cc:1347] Generated new cluster ID: 33d1be7d39fe40c3916372e11df474f9
I20260925 12:11:54.754333 5585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:54.754365 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.748077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.0.98.62:44095)
0925 12:11:54.748274 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.748282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.748307 (+ 25us) server_negotiation.cc:407] Connection header received
0925 12:11:54.748928 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.748936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.748998 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.749112 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:54.749489 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.750636 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.751846 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.752432 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.752935 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:54.753497 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:54.753501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:54.753501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:54.753744 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:54.753779 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:54.753873 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:54.753951 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:54.754044 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 12:11:54.754129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260925 12:11:54.755316 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:11:54.788480 5585 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:54.789242 5585 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:54.795782 5585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 777a27920ebc41e0be00418264afcfb7: Generated new TSK 0
I20260925 12:11:54.795995 5585 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 12:11:54.826953 5606 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 12:11:54.827085 5606 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 12:11:54.827627 5605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:54.804115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.0.1:32852)
0925 12:11:54.804318 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:54.804329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:54.804420 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:54.804679 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:54.804680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:54.804682 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:11:54.804754 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:54.804756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.805408 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.805409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:54.806413 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:54.806416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.806766 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.806769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:54.806840 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:54.806862 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:11:54.827193 (+ 20331us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 12:11:54.827293 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 12:11:54.827962 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:54.804176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.0.98.62:44095)
0925 12:11:54.804246 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:54.804248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:54.804399 (+ 151us) server_negotiation.cc:407] Connection header received
0925 12:11:54.804571 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:54.804572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:54.804584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:54.804647 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:11:54.804824 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.805319 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.806509 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:54.806692 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:54.806748 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:54.806967 (+ 219us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:11:54.827036 (+ 20069us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 12:11:54.827892 (+ 856us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 12:11:54.828016 5580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260925 12:11:54.828002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5509
2026-09-25T12:11:54Z chronyd exiting
I20260925 12:11:54.864225 392 test_util.cc:175] -----------------------------------------------
I20260925 12:11:54.864372 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790338290997044-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (252 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 12:11:54.865245 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:11:54.865947 392 webserver.cc:460] Webserver started at https://127.0.0.1:36019/ using document root <none> and password file <none>
I20260925 12:11:54.866014 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:11:54.866186 392 webserver.cc:460] Webserver started at http://127.0.0.1:36499/ using document root <none> and password file <none>
2026-09-25T12:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:54Z Disabled control of system clock
I20260925 12:11:54.873953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35317
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36019/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/testchainca.cert with env {}
W20260925 12:11:54.954118 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:54.954327 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:54.954351 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:54.955528 5612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:54.955574 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:54.955586 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:54.955598 5612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:54.955610 5612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:11:54.955660 5612 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:11:54.955677 5612 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36019/jwks.json
I20260925 12:11:54.957623 5612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36019/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:54.957903 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:54.958127 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:54.963147 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:54.963140 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.963284 5612 server_base.cc:1034] running on GCE node
W20260925 12:11:54.963140 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:54.963624 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:54.963965 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:54.965153 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790338314965146 us; error 58 us; skew 500 ppm
I20260925 12:11:54.971642 5612 webserver.cc:460] Webserver started at http://127.0.98.62:43709/ using document root <none> and password file <none>
I20260925 12:11:54.971938 5612 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:54.972018 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:54.972169 5612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:54.973364 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "7f700afea9554022872e893e2b87713b"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6575e786673991224e4feb4fc7e33202"
server_key_iv: "9e4da36ad24e0a369471913a09edd926"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.973724 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f700afea9554022872e893e2b87713b"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6575e786673991224e4feb4fc7e33202"
server_key_iv: "9e4da36ad24e0a369471913a09edd926"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.975117 5612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:54.975935 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:54.976157 5612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:54.976275 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "7f700afea9554022872e893e2b87713b"
format_stamp: "Formatted at 2026-09-25 12:11:54 on dist-test-slave-2hqq"
server_key: "6575e786673991224e4feb4fc7e33202"
server_key_iv: "9e4da36ad24e0a369471913a09edd926"
server_key_version: "encryptionkey@0"
I20260925 12:11:54.976372 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:54.992242 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:54.992672 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:54.992844 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:54.996656 5612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35317
I20260925 12:11:54.996695 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35317 every 8 connection(s)
I20260925 12:11:54.997053 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:54.998045 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:54.999133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5612
I20260925 12:11:54.999270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:54.999486 392 external_mini_cluster.cc:1428] Setting key 4f5fcdac4d13bb086465c165edc91828
I20260925 12:11:55.001402 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b: Bootstrap starting.
I20260925 12:11:55.002017 5679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:55.002301 5679 log.cc:824] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:55.003002 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b: No bootstrap required, opened a new log
I20260925 12:11:55.005038 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.000000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.0.98.62:35317)
0925 12:11:55.000220 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.000228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.000253 (+ 25us) server_negotiation.cc:407] Connection header received
0925 12:11:55.000765 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.000773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.000839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.000989 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:55.001242 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.002220 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.003057 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.003489 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.003622 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.004347 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:55.004351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:55.004352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:55.004569 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:55.004611 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:55.004623 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:55.004696 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:55.004771 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 12:11:55.004850 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260925 12:11:55.005164 5679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } }
I20260925 12:11:55.005357 5679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:55.005405 5679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f700afea9554022872e893e2b87713b, State: Initialized, Role: FOLLOWER
I20260925 12:11:55.005542 5679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } }
I20260925 12:11:55.005627 5679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:55.005694 5679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:55.005753 5679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:55.005924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:11:55.006404 5679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } }
I20260925 12:11:55.006523 5679 leader_election.cc:302] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f700afea9554022872e893e2b87713b; no voters:
I20260925 12:11:55.006700 5679 leader_election.cc:288] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:55.006731 5684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:55.006858 5684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [term 1 LEADER]: Becoming Leader. State: Replica: 7f700afea9554022872e893e2b87713b, State: Running, Role: LEADER
I20260925 12:11:55.006945 5684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } }
I20260925 12:11:55.006982 5679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:55.007234 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f700afea9554022872e893e2b87713b. Latest consensus state: current_term: 1 leader_uuid: "7f700afea9554022872e893e2b87713b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } } }
I20260925 12:11:55.007323 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.007244 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f700afea9554022872e893e2b87713b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f700afea9554022872e893e2b87713b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35317 } } }
I20260925 12:11:55.007448 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.007548 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:55.007984 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:55.013577 5688 catalog_manager.cc:1347] Generated new cluster ID: 3d92737e7e21443aa9fc33d4aabfa5fc
I20260925 12:11:55.013670 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:55.028889 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:55.029639 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:55.037971 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f700afea9554022872e893e2b87713b: Generated new TSK 0
I20260925 12:11:55.038188 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 12:11:55.063834 5709 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 12:11:55.063967 5709 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 12:11:55.064296 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.043747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.0.98.62:35317)
0925 12:11:55.043792 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.043794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.043879 (+ 85us) server_negotiation.cc:407] Connection header received
0925 12:11:55.043968 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.043969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.043979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.044039 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:11:55.044191 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.044777 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.045941 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.046147 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.046214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.046415 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:11:55.064038 (+ 17623us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 12:11:55.064205 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 12:11:55.064359 5682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 12:11:55.064406 5708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:55.043650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35317 (local address 127.0.0.1:39718)
0925 12:11:55.043817 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:55.043828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:55.043899 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:55.044065 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:55.044065 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:55.044067 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:11:55.044126 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.044128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.044866 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.044867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.045829 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.045832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.046238 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.046242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.046306 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.046312 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:11:55.064169 (+ 17857us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 12:11:55.064291 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260925 12:11:55.064854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5612
2026-09-25T12:11:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 12:11:55.102670 392 mini_oidc.cc:152] Starting JWKS server
I20260925 12:11:55.103365 392 webserver.cc:460] Webserver started at https://127.0.0.1:42317/ using document root <none> and password file <none>
I20260925 12:11:55.103435 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 12:11:55.103576 392 webserver.cc:460] Webserver started at http://127.0.0.1:32879/ using document root <none> and password file <none>
2026-09-25T12:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:55Z Disabled control of system clock
I20260925 12:11:55.111073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43523
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43523
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 12:11:55.191694 5715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:55.191979 5715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:55.192021 5715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:55.193284 5715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:55.193359 5715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:55.193380 5715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:55.193401 5715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:55.193431 5715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:11:55.193455 5715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:11:55.193475 5715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42317/jwks.json
I20260925 12:11:55.194906 5715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:55.195201 5715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:55.195515 5715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:55.200470 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:55.200555 5723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:55.200453 5720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:55.200687 5715 server_base.cc:1034] running on GCE node
I20260925 12:11:55.200941 5715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:55.201227 5715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:55.202435 5715 hybrid_clock.cc:648] HybridClock initialized: now 1790338315202402 us; error 44 us; skew 500 ppm
I20260925 12:11:55.203730 5715 webserver.cc:460] Webserver started at http://127.0.98.62:38591/ using document root <none> and password file <none>
I20260925 12:11:55.203979 5715 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:55.204052 5715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:55.204200 5715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:55.205415 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "1aede32f8c2747fead30dcfc66703c84"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "a1a2e5ccbe2bac784a92408178114099"
server_key_iv: "e86b63e53100b50de010601fc4a9a5b8"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.205781 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "1aede32f8c2747fead30dcfc66703c84"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "a1a2e5ccbe2bac784a92408178114099"
server_key_iv: "e86b63e53100b50de010601fc4a9a5b8"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.207111 5715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:11:55.207818 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:55.208001 5715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:11:55.208104 5715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "1aede32f8c2747fead30dcfc66703c84"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "a1a2e5ccbe2bac784a92408178114099"
server_key_iv: "e86b63e53100b50de010601fc4a9a5b8"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.208177 5715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:55.219667 5715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:55.219962 5715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:55.220113 5715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:55.223754 5715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43523
I20260925 12:11:55.223832 5781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43523 every 8 connection(s)
I20260925 12:11:55.224161 5715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:55.224900 5782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:55.225169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5715
I20260925 12:11:55.225247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:55.225464 392 external_mini_cluster.cc:1428] Setting key 8b88cfe69401865260b86aab523b6ab3
I20260925 12:11:55.228404 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84: Bootstrap starting.
I20260925 12:11:55.229110 5782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:55.229400 5782 log.cc:824] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:55.230177 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84: No bootstrap required, opened a new log
I20260925 12:11:55.231161 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.226052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.0.98.62:43523)
0925 12:11:55.226256 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.226263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.226288 (+ 25us) server_negotiation.cc:407] Connection header received
0925 12:11:55.226754 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.226761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.226826 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.226945 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:55.227166 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.228066 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.229199 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.229658 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.229787 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.230489 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:55.230493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:55.230493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:55.230650 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:55.230681 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:55.230691 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:55.230744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:55.230857 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:11:55.230950 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260925 12:11:55.232121 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:11:55.232424 5782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } }
I20260925 12:11:55.232635 5782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:55.232683 5782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aede32f8c2747fead30dcfc66703c84, State: Initialized, Role: FOLLOWER
I20260925 12:11:55.232806 5782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } }
I20260925 12:11:55.232892 5782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:55.232932 5782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:55.232986 5782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:55.233557 5782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } }
I20260925 12:11:55.233671 5782 leader_election.cc:302] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aede32f8c2747fead30dcfc66703c84; no voters:
I20260925 12:11:55.233829 5782 leader_election.cc:288] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:55.233911 5787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:55.234050 5787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [term 1 LEADER]: Becoming Leader. State: Replica: 1aede32f8c2747fead30dcfc66703c84, State: Running, Role: LEADER
I20260925 12:11:55.234140 5787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } }
I20260925 12:11:55.234149 5782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:55.234525 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aede32f8c2747fead30dcfc66703c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } } }
I20260925 12:11:55.234535 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aede32f8c2747fead30dcfc66703c84. Latest consensus state: current_term: 1 leader_uuid: "1aede32f8c2747fead30dcfc66703c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aede32f8c2747fead30dcfc66703c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43523 } } }
I20260925 12:11:55.234633 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.234649 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.234877 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:55.235345 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:55.236866 5794 catalog_manager.cc:1347] Generated new cluster ID: 13f9f7d6dc7d4e3abf726eb8ae59b19e
I20260925 12:11:55.236946 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:55.249285 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:55.249985 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:55.254842 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aede32f8c2747fead30dcfc66703c84: Generated new TSK 0
I20260925 12:11:55.255020 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:11:55.266741 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.262824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.98.62:43523)
0925 12:11:55.262886 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.262888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.263047 (+ 159us) server_negotiation.cc:407] Connection header received
0925 12:11:55.263111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.263112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.263126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.263211 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:11:55.263325 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.264131 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.265000 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.265313 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.265385 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.265537 (+ 152us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:11:55.266519 (+ 982us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 12:11:55.266651 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 12:11:55.266814 5785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 12:11:55.268697 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:55.262742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43523 (local address 127.0.0.1:47350)
0925 12:11:55.262971 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:55.262982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:55.263052 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:55.263214 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:55.263215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:55.263216 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:11:55.263265 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.263267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.264213 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.264214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.264920 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.264923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.265393 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.265398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.265437 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.265442 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:11:55.266613 (+ 1171us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 12:11:55.268603 (+ 1990us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1765,"thread_start_us":184,"threads_started":1}
I20260925 12:11:55.269120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5715
E20260925 12:11:55.279268 5812 webserver.cc:533] Webserver: sslize failed:
E20260925 12:11:55.279340 5812 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T12:11:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T12:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:55Z Disabled control of system clock
I20260925 12:11:55.314431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41837
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41837
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/testchainca.cert with env {}
W20260925 12:11:55.393954 5816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:55.394158 5816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:55.394179 5816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:55.395244 5816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:55.395278 5816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:55.395290 5816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:55.395303 5816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:55.395315 5816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 12:11:55.395327 5816 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 12:11:55.395339 5816 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 12:11:55.396764 5816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:55.396960 5816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:55.397118 5816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:55.401908 5824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:55.401939 5821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:55.402032 5816 server_base.cc:1034] running on GCE node
W20260925 12:11:55.401911 5822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:55.402366 5816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:55.402681 5816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:55.403873 5816 hybrid_clock.cc:648] HybridClock initialized: now 1790338315403884 us; error 53 us; skew 500 ppm
I20260925 12:11:55.405161 5816 webserver.cc:460] Webserver started at http://127.0.98.62:38721/ using document root <none> and password file <none>
I20260925 12:11:55.405431 5816 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:55.405503 5816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:55.405648 5816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:55.406749 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "7a0bef7c8c5c438209578a624a3248b5"
server_key_iv: "71e386efd5afe0d3af2d4204ff50bf77"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.407119 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "7a0bef7c8c5c438209578a624a3248b5"
server_key_iv: "71e386efd5afe0d3af2d4204ff50bf77"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.408388 5816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:11:55.409240 5830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:55.409415 5816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:11:55.409511 5816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0"
format_stamp: "Formatted at 2026-09-25 12:11:55 on dist-test-slave-2hqq"
server_key: "7a0bef7c8c5c438209578a624a3248b5"
server_key_iv: "71e386efd5afe0d3af2d4204ff50bf77"
server_key_version: "encryptionkey@0"
I20260925 12:11:55.409592 5816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:55.422503 5816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:55.422839 5816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:55.422997 5816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:55.427014 5816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41837
I20260925 12:11:55.427045 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41837 every 8 connection(s)
I20260925 12:11:55.427382 5816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:55.428344 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:55.428682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5816
I20260925 12:11:55.428865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:55.429069 392 external_mini_cluster.cc:1428] Setting key 5021c556a67669a8237da0486018629f
I20260925 12:11:55.431998 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0: Bootstrap starting.
I20260925 12:11:55.432677 5883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:55.433019 5883 log.cc:824] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:55.433772 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0: No bootstrap required, opened a new log
I20260925 12:11:55.435088 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.429590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.0.98.62:41837)
0925 12:11:55.429796 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.429805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.429838 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:11:55.430333 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.430343 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.430423 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.430574 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:55.430732 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.432083 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.432967 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.433500 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.433624 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.434347 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:55.434352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:55.434352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:11:55.434554 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:55.434595 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:11:55.434611 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:55.434685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:55.434789 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:11:55.434864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260925 12:11:55.436206 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:11:55.436357 5883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } }
I20260925 12:11:55.436540 5883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:55.436581 5883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e2ec58d6a4480d8c5bdc10fbccd1d0, State: Initialized, Role: FOLLOWER
I20260925 12:11:55.436700 5883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } }
I20260925 12:11:55.436777 5883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:55.436807 5883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:55.436846 5883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:55.437546 5883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } }
I20260925 12:11:55.437636 5883 leader_election.cc:302] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68e2ec58d6a4480d8c5bdc10fbccd1d0; no voters:
I20260925 12:11:55.437760 5883 leader_election.cc:288] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:55.437825 5888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:55.437970 5888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 68e2ec58d6a4480d8c5bdc10fbccd1d0, State: Running, Role: LEADER
I20260925 12:11:55.438020 5883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:55.438058 5888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } }
I20260925 12:11:55.438421 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } } }
I20260925 12:11:55.438558 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.438757 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68e2ec58d6a4480d8c5bdc10fbccd1d0. Latest consensus state: current_term: 1 leader_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e2ec58d6a4480d8c5bdc10fbccd1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41837 } } }
I20260925 12:11:55.438829 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:55.439105 5897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:55.439590 5897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:55.441296 5897 catalog_manager.cc:1347] Generated new cluster ID: 269a28edc08a4cce8693d8ffe5127a38
I20260925 12:11:55.441354 5897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:55.460672 5897 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:55.461419 5897 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:55.470979 5897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68e2ec58d6a4480d8c5bdc10fbccd1d0: Generated new TSK 0
I20260925 12:11:55.471201 5897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:11:55.674029 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:55.481261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.0.98.62:41837)
0925 12:11:55.481323 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:55.481325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:55.481426 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:11:55.481484 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:55.481485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:55.481498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:55.481581 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0925 12:11:55.481705 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.482506 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.483276 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.483592 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.483663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.483817 (+ 154us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 12:11:55.673801 (+189984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 12:11:55.673968 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36698: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 12:11:55.674142 5886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36698: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 12:11:55.674147 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:55.481168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41837 (local address 127.0.0.1:36698)
0925 12:11:55.481348 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:55.481359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:55.481423 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:55.481587 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:55.481587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:55.481589 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 12:11:55.481643 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.481644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.482610 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.482610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.483194 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:55.483196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:55.483666 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:55.483672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:55.483711 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:55.483715 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 12:11:55.673947 (+190232us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 12:11:55.674073 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260925 12:11:55.674574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5816
2026-09-25T12:11:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (384 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:55 dist-test-slave-2hqq krb5kdc[5924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:55 dist-test-slave-2hqq krb5kdc[5924](info): set up 2 sockets
Sep 25 12:11:55 dist-test-slave-2hqq krb5kdc[5924](info): commencing operation
krb5kdc: starting...
W20260925 12:11:56.888934 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:56 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:11:57.007145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:57.087092 5937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:57.087356 5937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:57.087394 5937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:57.088593 5937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:57.088673 5937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:57.088699 5937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:57.088727 5937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:57.088757 5937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:57.090278 5937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:57.090554 5937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:57.090785 5937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:57.095764 5942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:57.095770 5945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.095979 5937 server_base.cc:1034] running on GCE node
W20260925 12:11:57.095777 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.096305 5937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:57.096624 5937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:57.097786 5937 hybrid_clock.cc:648] HybridClock initialized: now 1790338317097773 us; error 29 us; skew 500 ppm
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.100246 5937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:11:57.100788 5937 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260925 12:11:57.101018 5937 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:57.101085 5937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:57.101241 5937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:57.102313 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "4741f77a1ba4496c857396a2565c6826"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "895b97041368714382be95e110b3aa54"
server_key_iv: "53d83649ddb56df0a70d688b762b3566"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.102690 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "4741f77a1ba4496c857396a2565c6826"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "895b97041368714382be95e110b3aa54"
server_key_iv: "53d83649ddb56df0a70d688b762b3566"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.104018 5937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:57.104831 5952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.105001 5937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:57.105105 5937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "4741f77a1ba4496c857396a2565c6826"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "895b97041368714382be95e110b3aa54"
server_key_iv: "53d83649ddb56df0a70d688b762b3566"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.105183 5937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:57.128818 5937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:57.130663 5937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:57.130831 5937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:57.134925 5937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45315
I20260925 12:11:57.134969 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45315 every 8 connection(s)
I20260925 12:11:57.135474 5937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:57.136133 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:57.138592 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826: Bootstrap starting.
I20260925 12:11:57.139148 6005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:57.139401 6005 log.cc:824] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:57.140057 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826: No bootstrap required, opened a new log
I20260925 12:11:57.141654 6005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260925 12:11:57.141788 6005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:57.141860 6005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4741f77a1ba4496c857396a2565c6826, State: Initialized, Role: FOLLOWER
I20260925 12:11:57.142038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 5937
I20260925 12:11:57.142024 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260925 12:11:57.142113 6005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:57.142150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:57.142150 6005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:57.142208 6005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:11:57.142352 392 external_mini_cluster.cc:1428] Setting key a371bd2e39425b69a894bfcb3a99807e
I20260925 12:11:57.143396 6005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260925 12:11:57.143512 6005 leader_election.cc:302] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4741f77a1ba4496c857396a2565c6826; no voters:
I20260925 12:11:57.143690 6005 leader_election.cc:288] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:57.143714 6010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:57.143869 6010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [term 1 LEADER]: Becoming Leader. State: Replica: 4741f77a1ba4496c857396a2565c6826, State: Running, Role: LEADER
I20260925 12:11:57.143946 6005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:57.143956 6010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260925 12:11:57.144277 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4741f77a1ba4496c857396a2565c6826. Latest consensus state: current_term: 1 leader_uuid: "4741f77a1ba4496c857396a2565c6826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260925 12:11:57.144348 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:57.144575 6016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:57.145267 6016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:57.145529 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4741f77a1ba4496c857396a2565c6826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741f77a1ba4496c857396a2565c6826" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260925 12:11:57.145869 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:57.146708 6016 catalog_manager.cc:1347] Generated new cluster ID: 763e41fed2424fa5b569603a050848f7
I20260925 12:11:57.146757 6016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:57.153872 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.142825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.0.98.62:45315)
0925 12:11:57.143010 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.143019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.143053 (+ 34us) server_negotiation.cc:407] Connection header received
0925 12:11:57.143156 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.143161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.143202 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.143310 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.144212 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.145658 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.146664 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.147243 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.147342 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.150115 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.150119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.150119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.150131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.152084 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.152586 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.152590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.152591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.152738 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.152953 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.152954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.152954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.153277 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.153451 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.153520 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.153669 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260925 12:11:57.161062 6016 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:57.161598 6016 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:57.164351 6016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4741f77a1ba4496c857396a2565c6826: Generated new TSK 0
I20260925 12:11:57.164549 6016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:11:57.198764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:57.278916 6031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:57.279120 6031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:57.279143 6031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:57.280279 6031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:57.280326 6031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:57.280351 6031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:57.281805 6031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:57.282044 6031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:57.282279 6031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:57.287302 6037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:57.287273 6039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.287436 6031 server_base.cc:1034] running on GCE node
W20260925 12:11:57.287285 6036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.287690 6031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:57.287981 6031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:57.289129 6031 hybrid_clock.cc:648] HybridClock initialized: now 1790338317289118 us; error 34 us; skew 500 ppm
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.291460 6031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:11:57.291909 6031 webserver.cc:460] Webserver started at http://127.0.98.1:46757/ using document root <none> and password file <none>
I20260925 12:11:57.292173 6031 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:57.292220 6031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:57.292306 6031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:57.293359 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "82bca99210124e8ea4dc71b6cc1bbed2"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "c6433b93a7d7db2f3e0daf6becc6a5ef"
server_key_iv: "4c24fdd6aad8c9ddc06462f5f0d2742b"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.293654 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82bca99210124e8ea4dc71b6cc1bbed2"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "c6433b93a7d7db2f3e0daf6becc6a5ef"
server_key_iv: "4c24fdd6aad8c9ddc06462f5f0d2742b"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.294878 6031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:11:57.295542 6046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.295748 6031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:11:57.295840 6031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "82bca99210124e8ea4dc71b6cc1bbed2"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "c6433b93a7d7db2f3e0daf6becc6a5ef"
server_key_iv: "4c24fdd6aad8c9ddc06462f5f0d2742b"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.295893 6031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:57.304140 6031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:57.305939 6031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:57.306046 6031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:57.306223 6031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:57.306466 6031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:57.306495 6031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.306515 6031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:57.306528 6031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.313025 6031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34515
I20260925 12:11:57.313126 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34515 every 8 connection(s)
I20260925 12:11:57.313381 6031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:57.322110 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.314061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49107 (local address 127.0.98.62:45315)
0925 12:11:57.314131 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.314133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.314328 (+ 195us) server_negotiation.cc:407] Connection header received
0925 12:11:57.315245 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.315246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.315255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.315319 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.315859 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.316366 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.317368 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.317566 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.317654 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.319733 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.319734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.319734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.319741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.320975 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.321340 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.321341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.321341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.321391 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.321606 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.321606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.321606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.321704 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.321757 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.321979 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.322042 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:11:57.322304 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.314025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:49107)
0925 12:11:57.314201 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:57.314241 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:57.315158 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:57.315373 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:57.315376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:57.315615 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:57.315794 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.315797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.316440 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.316441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.317298 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.317300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.317673 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.317674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.317798 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.318318 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:57.318326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:57.319647 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:57.321044 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.321050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.321052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.321260 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.321451 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.321451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.321451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.321542 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.321760 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:57.321764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:57.321871 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 12:11:57.321950 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 12:11:57.322057 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260925 12:11:57.322796 6160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260925 12:11:57.322899 6160 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:57.323063 6160 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260925 12:11:57.323444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6031
I20260925 12:11:57.323547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:57.323673 5969 ts_manager.cc:194] Registered new tserver with Master: 82bca99210124e8ea4dc71b6cc1bbed2 (127.0.98.1:34515)
I20260925 12:11:57.323776 392 external_mini_cluster.cc:1428] Setting key ec6911b98dfdf10514278541c6ec8fc5
I20260925 12:11:57.324366 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49107
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:11:57.347666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:57.428548 6165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:57.428753 6165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:57.428777 6165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:57.429838 6165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:57.429872 6165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:57.429898 6165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:11:57.431279 6165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:57.431463 6165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:57.431713 6165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:57.436837 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.436898 6165 server_base.cc:1034] running on GCE node
W20260925 12:11:57.436846 6170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:57.436854 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.437255 6165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:57.437458 6165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:57.438601 6165 hybrid_clock.cc:648] HybridClock initialized: now 1790338317438579 us; error 45 us; skew 500 ppm
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.441018 6165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:11:57.441478 6165 webserver.cc:460] Webserver started at http://127.0.98.2:45167/ using document root <none> and password file <none>
I20260925 12:11:57.441738 6165 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:57.441803 6165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:57.441937 6165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:57.442993 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "32824e0f8a864799ab237a132d7c4eef"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "98ea3339f104b8b703bbdf7289b0accb"
server_key_iv: "ebb5daf4183bf16bb6c9df6dd90ed8e2"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.443357 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32824e0f8a864799ab237a132d7c4eef"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "98ea3339f104b8b703bbdf7289b0accb"
server_key_iv: "ebb5daf4183bf16bb6c9df6dd90ed8e2"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.444911 6165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:57.445842 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.446028 6165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:57.446134 6165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "32824e0f8a864799ab237a132d7c4eef"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "98ea3339f104b8b703bbdf7289b0accb"
server_key_iv: "ebb5daf4183bf16bb6c9df6dd90ed8e2"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.446225 6165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:57.452801 6165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:57.454592 6165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:57.454749 6165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:57.454963 6165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:57.455257 6165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:57.455312 6165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.455353 6165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:57.455380 6165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.461063 6165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39893
I20260925 12:11:57.461105 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39893 every 8 connection(s)
I20260925 12:11:57.461452 6165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:11:57.462183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6165
I20260925 12:11:57.462281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:11:57.462497 392 external_mini_cluster.cc:1428] Setting key b2c01913db2e929d2991f558a39a86e1
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:11:57.471472 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.462199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60685 (local address 127.0.98.62:45315)
0925 12:11:57.462269 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.462272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.462506 (+ 234us) server_negotiation.cc:407] Connection header received
0925 12:11:57.463590 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.463591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.463603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.463671 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.464242 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.464886 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.465904 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.466158 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.466219 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.468423 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.468425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.468425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.468432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.469967 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.470486 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.470487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.470488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.470557 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.470807 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.470808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.470808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.470928 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.471000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.471300 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.471376 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:11:57.471670 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.462161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:60685)
0925 12:11:57.462371 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:57.462410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:57.463490 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:57.463729 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:57.463732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:57.463975 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:57.464156 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.464159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.464988 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.464990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.465799 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.465801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.466224 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.466226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.466363 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.466870 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:57.466906 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:57.468320 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:57.470095 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.470102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.470104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.470394 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.470625 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.470626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.470626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.470738 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.471005 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:57.471018 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:57.471172 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 12:11:57.471282 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 12:11:57.471439 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260925 12:11:57.472154 6294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260925 12:11:57.472235 6294 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:57.472437 6294 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 12:11:57.472882 5969 ts_manager.cc:194] Registered new tserver with Master: 32824e0f8a864799ab237a132d7c4eef (127.0.98.2:39893)
I20260925 12:11:57.473256 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:11:57.488001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:57.569655 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:57.569870 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:57.569893 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:57.571033 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:57.571082 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:57.571108 6299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:11:57.572654 6299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:57.572896 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:57.573139 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:57.578044 6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:57.578117 6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:57.578048 6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:57.578110 6299 server_base.cc:1034] running on GCE node
I20260925 12:11:57.578452 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:57.578694 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:57.579854 6299 hybrid_clock.cc:648] HybridClock initialized: now 1790338317579829 us; error 39 us; skew 500 ppm
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.582443 6299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:11:57.582907 6299 webserver.cc:460] Webserver started at http://127.0.98.3:33809/ using document root <none> and password file <none>
I20260925 12:11:57.583144 6299 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:57.583212 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:57.583359 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:57.584429 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "237c96e583d54637b4bf91b41e730166"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "7762e735e8501fa6dec35d49047eac86"
server_key_iv: "2cc473ce687d00a626adae6ab3e988d0"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.584893 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "237c96e583d54637b4bf91b41e730166"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "7762e735e8501fa6dec35d49047eac86"
server_key_iv: "2cc473ce687d00a626adae6ab3e988d0"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.586194 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:57.586992 6314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.587204 6299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:57.587309 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "237c96e583d54637b4bf91b41e730166"
format_stamp: "Formatted at 2026-09-25 12:11:57 on dist-test-slave-2hqq"
server_key: "7762e735e8501fa6dec35d49047eac86"
server_key_iv: "2cc473ce687d00a626adae6ab3e988d0"
server_key_version: "encryptionkey@0"
I20260925 12:11:57.587395 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:57.601066 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:57.602936 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:57.603044 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:57.603212 6299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:57.603530 6299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:57.603603 6299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.603657 6299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:57.603690 6299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:57.609584 6299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39225
I20260925 12:11:57.609622 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39225 every 8 connection(s)
I20260925 12:11:57.609982 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:11:57.612732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6299
I20260925 12:11:57.612861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:11:57.613040 392 external_mini_cluster.cc:1428] Setting key 5d48cd1fc27a358cf4e977632e5486ac
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:11:57.619555 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.610719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43923 (local address 127.0.98.62:45315)
0925 12:11:57.610778 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.610780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.610971 (+ 191us) server_negotiation.cc:407] Connection header received
0925 12:11:57.611889 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.611890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.611900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.611962 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.612684 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.613328 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.614329 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.614535 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.614611 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.616788 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.616790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.616790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.616797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.618161 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.618589 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.618590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.618591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.618650 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.618907 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.618908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.618908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.619022 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.619100 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.619384 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.619466 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:11:57.619743 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.610673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:43923)
0925 12:11:57.610832 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:57.610879 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:57.611794 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:57.612055 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:57.612059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:57.612308 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:57.612604 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.612610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.613422 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.613423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.614223 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:57.614224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.614616 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.614617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:57.614748 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.615304 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:57.615312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:57.616688 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:57.618261 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.618268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.618271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.618496 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.618716 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:57.618716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:57.618717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:57.618830 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:57.619103 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:57.619109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:57.619250 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 12:11:57.619351 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:11:57.619501 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260925 12:11:57.620254 6428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260925 12:11:57.620334 6428 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:57.620625 6428 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260925 12:11:57.621017 5969 ts_manager.cc:194] Registered new tserver with Master: 237c96e583d54637b4bf91b41e730166 (127.0.98.3:39225)
I20260925 12:11:57.621371 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43923
I20260925 12:11:57.624845 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/test-admin.keytab.
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.640245 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 12:11:57.641917 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:57.641170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.1:34515)
0925 12:11:57.641335 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.641341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.641355 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:11:57.641385 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.641387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.641400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.641466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.641825 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34966: BlockingRecv error: recv got EOF from 127.0.0.1:34966 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
W20260925 12:11:57.642007 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.643052 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.643954 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:57.651818 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.644624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.1:34515)
0925 12:11:57.644687 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.644689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.644696 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:11:57.644717 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.644718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.644726 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.644776 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.645236 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.646081 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.646781 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.647114 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.647175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.649268 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.649275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.649276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.649282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.650871 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.651174 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.651176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.651177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.651228 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.651384 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.651384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.651385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.651503 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.651612 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.651689 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.651745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 12:11:57.653677 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:57.653124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.0.98.1:34515)
0925 12:11:57.653200 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.653201 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.653240 (+ 39us) server_negotiation.cc:407] Connection header received
0925 12:11:57.653287 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.653288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.653296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.653340 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.653627 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34988: BlockingRecv error: recv got EOF from 127.0.0.1:34988 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 12:11:57.653797 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc)
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.654778 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.655623 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:57.662379 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.656105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.1:34515)
0925 12:11:57.656185 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.656186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.656230 (+ 44us) server_negotiation.cc:407] Connection header received
0925 12:11:57.656267 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.656268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.656276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.656319 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.656743 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.657220 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.657868 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.658053 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.658101 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.660059 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.660061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.660061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.660067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.661408 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.661722 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.661723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.661723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.661800 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.661959 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.661960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.661960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.662043 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.662109 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.662192 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.662244 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 12:11:57.663836 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:11:57.663342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.98.1:34515)
0925 12:11:57.663398 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.663399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.663425 (+ 26us) server_negotiation.cc:407] Connection header received
0925 12:11:57.663471 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.663472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.663478 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.663520 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.663806 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34994: BlockingRecv error: recv got EOF from 127.0.0.1:34994 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 12:11:57.663911 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790338290997044-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.665148 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:57.666064 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:11:57.673000 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:57.666603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.1:34515)
0925 12:11:57.666659 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:57.666661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:57.666764 (+ 103us) server_negotiation.cc:407] Connection header received
0925 12:11:57.666815 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:57.666816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:57.666823 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:57.666861 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:57.667213 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.667688 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.668363 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:57.668587 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:57.668641 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:57.670557 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:57.670558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:57.670558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:57.670564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:57.671931 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.672231 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.672232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.672232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.672278 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:57.672548 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:57.672549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:57.672550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:57.672646 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:57.672717 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:57.672826 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 12:11:57.672878 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:11:57.673705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6031
I20260925 12:11:57.678305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6165
I20260925 12:11:57.682384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6299
I20260925 12:11:57.687489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 5937
2026-09-25T12:11:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1839 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[6448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[6448](info): set up 2 sockets
Sep 25 12:11:57 dist-test-slave-2hqq krb5kdc[6448](info): commencing operation
krb5kdc: starting...
W20260925 12:11:58.718528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:11:58 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:11:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 12:11:58.847684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:58.926746 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:58.926959 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:58.926980 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:58.928061 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:11:58.928093 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:58.928107 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:58.928119 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:11:58.928131 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:11:58.929558 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:58.929749 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:58.929914 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:58.934823 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:58.934864 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:58.934830 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:58.934863 6461 server_base.cc:1034] running on GCE node
I20260925 12:11:58.935321 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:58.935612 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:58.936769 6461 hybrid_clock.cc:648] HybridClock initialized: now 1790338318936747 us; error 37 us; skew 500 ppm
Sep 25 12:11:58 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:58.939105 6461 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 12:11:58.939587 6461 webserver.cc:460] Webserver started at http://127.0.98.62:34645/ using document root <none> and password file <none>
I20260925 12:11:58.939814 6461 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:58.939883 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:58.940038 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:58.941151 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "bba85b6019884f15a1119d6a183f1763"
format_stamp: "Formatted at 2026-09-25 12:11:58 on dist-test-slave-2hqq"
server_key: "a073f1b204dfb226f95d1c3e72ffcf5b"
server_key_iv: "d8d46d8bf0b924b2f58323fc1823bff1"
server_key_version: "encryptionkey@0"
I20260925 12:11:58.941511 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "bba85b6019884f15a1119d6a183f1763"
format_stamp: "Formatted at 2026-09-25 12:11:58 on dist-test-slave-2hqq"
server_key: "a073f1b204dfb226f95d1c3e72ffcf5b"
server_key_iv: "d8d46d8bf0b924b2f58323fc1823bff1"
server_key_version: "encryptionkey@0"
I20260925 12:11:58.942832 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:11:58.943665 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:58.943849 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:58.943958 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "bba85b6019884f15a1119d6a183f1763"
format_stamp: "Formatted at 2026-09-25 12:11:58 on dist-test-slave-2hqq"
server_key: "a073f1b204dfb226f95d1c3e72ffcf5b"
server_key_iv: "d8d46d8bf0b924b2f58323fc1823bff1"
server_key_version: "encryptionkey@0"
I20260925 12:11:58.944034 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:58.963986 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:58.966060 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:58.966229 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:58.970491 6461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40957
I20260925 12:11:58.970530 6528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40957 every 8 connection(s)
I20260925 12:11:58.970944 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:11:58.971984 6529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:11:58.972311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6461
I20260925 12:11:58.972407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:11:58.972698 392 external_mini_cluster.cc:1428] Setting key 8a59db982ef5980cd377361458d5e571
I20260925 12:11:58.975147 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Bootstrap starting.
I20260925 12:11:58.975770 6529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Neither blocks nor log segments found. Creating new log.
I20260925 12:11:58.976048 6529 log.cc:824] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Log is configured to *not* fsync() on all Append() calls
I20260925 12:11:58.976789 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: No bootstrap required, opened a new log
I20260925 12:11:58.978816 6529 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } }
I20260925 12:11:58.979013 6529 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:11:58.979058 6529 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba85b6019884f15a1119d6a183f1763, State: Initialized, Role: FOLLOWER
I20260925 12:11:58.979193 6529 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } }
I20260925 12:11:58.979279 6529 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:11:58.979323 6529 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:11:58.979374 6529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 12:11:58 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:11:58.980110 6529 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } }
I20260925 12:11:58.980201 6529 leader_election.cc:302] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba85b6019884f15a1119d6a183f1763; no voters:
I20260925 12:11:58.980365 6529 leader_election.cc:288] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:11:58.980427 6534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:11:58.980592 6534 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 LEADER]: Becoming Leader. State: Replica: bba85b6019884f15a1119d6a183f1763, State: Running, Role: LEADER
I20260925 12:11:58.980687 6529 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:11:58.980676 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } }
I20260925 12:11:58.981020 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba85b6019884f15a1119d6a183f1763. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } }
I20260925 12:11:58.981102 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
I20260925 12:11:58.981541 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } }
I20260925 12:11:58.981611 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
W20260925 12:11:58.982856 6549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:11:58.982935 6549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:11:58.983014 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:11:58.983220 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:11:58.984295 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:58.973259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.0.98.62:40957)
0925 12:11:58.973441 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:58.973449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:58.973479 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:11:58.973875 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:58.973881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:58.973927 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:58.974047 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:58.974869 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:58.975809 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:58.976839 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:58.977287 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:58.977508 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:58.980101 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:58.980105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:58.980105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:58.980115 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:58.982570 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:58.983054 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:58.983056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:58.983056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:58.983156 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:58.983371 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:58.983372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:58.983372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:58.983693 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:58.983848 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:58.983965 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 12:11:58.984101 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260925 12:11:58.984802 6550 catalog_manager.cc:1347] Generated new cluster ID: 1a8051be89a64c51810eaa51b3efcbb6
I20260925 12:11:58.984850 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:11:59.005395 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:11:59.005950 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:11:59.008838 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Generated new TSK 0
I20260925 12:11:59.009034 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 12:11:59.039683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:11:59.144904 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:11:59.145118 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:11:59.145143 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:11:59.146457 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:11:59.146516 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:11:59.146605 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:11:59.148386 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:11:59.148690 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:11:59.148968 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:11:59.154378 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:11:59.154510 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:59.154409 6555 server_base.cc:1034] running on GCE node
W20260925 12:11:59.154390 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:11:59.154872 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:11:59.155171 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:11:59.156373 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790338319156348 us; error 34 us; skew 500 ppm
Sep 25 12:11:59 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338319, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:11:59.159392 6555 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:11:59.159838 6555 webserver.cc:460] Webserver started at http://127.0.98.1:34223/ using document root <none> and password file <none>
I20260925 12:11:59.160084 6555 fs_manager.cc:359] Metadata directory not provided
I20260925 12:11:59.160153 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:11:59.160303 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:11:59.161427 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc345b8b302144dab0ff926d26851f2f"
format_stamp: "Formatted at 2026-09-25 12:11:59 on dist-test-slave-2hqq"
server_key: "e9baad2a85e7c1be1eb3e35c8460062c"
server_key_iv: "cb4f05178f0d25f24508930197998012"
server_key_version: "encryptionkey@0"
I20260925 12:11:59.161794 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc345b8b302144dab0ff926d26851f2f"
format_stamp: "Formatted at 2026-09-25 12:11:59 on dist-test-slave-2hqq"
server_key: "e9baad2a85e7c1be1eb3e35c8460062c"
server_key_iv: "cb4f05178f0d25f24508930197998012"
server_key_version: "encryptionkey@0"
I20260925 12:11:59.163514 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:11:59.164472 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:59.164685 6555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:11:59.164776 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "fc345b8b302144dab0ff926d26851f2f"
format_stamp: "Formatted at 2026-09-25 12:11:59 on dist-test-slave-2hqq"
server_key: "e9baad2a85e7c1be1eb3e35c8460062c"
server_key_iv: "cb4f05178f0d25f24508930197998012"
server_key_version: "encryptionkey@0"
I20260925 12:11:59.164865 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:11:59.184690 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:11:59.186936 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:11:59.187095 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:11:59.187348 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:11:59.187670 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:11:59.187703 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:59.187726 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:11:59.187777 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:11:59.193926 6555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46681
I20260925 12:11:59.194024 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46681 every 8 connection(s)
I20260925 12:11:59.194345 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:11:59.195328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6555
I20260925 12:11:59.195436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:11:59.195633 392 external_mini_cluster.cc:1428] Setting key c3908700afcdeb943499c976ae4a2c06
Sep 25 12:11:59 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338319, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:11:59.205238 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:59.195125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45569 (local address 127.0.98.62:40957)
0925 12:11:59.195221 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:11:59.195224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:11:59.195400 (+ 176us) server_negotiation.cc:407] Connection header received
0925 12:11:59.196031 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:11:59.196032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:11:59.196043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:11:59.196089 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:11:59.197225 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:59.198074 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:59.199187 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:11:59.199557 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:11:59.199633 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:59.202132 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:11:59.202134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:11:59.202134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:11:59.202143 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:11:59.203709 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:59.204127 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:59.204129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:59.204129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:59.204190 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:11:59.204461 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:11:59.204462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:11:59.204462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:11:59.204635 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:11:59.204713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:11:59.205045 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 12:11:59.205141 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 12:11:59.205421 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:11:59.195088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:45569)
0925 12:11:59.195277 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:11:59.195319 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:11:59.195948 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:11:59.196164 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:11:59.196168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:11:59.196830 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:11:59.197098 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:59.197104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:59.198190 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:59.198191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:59.199078 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:11:59.199082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:11:59.199684 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:11:59.199685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:11:59.199793 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:11:59.200369 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:11:59.200378 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:11:59.202022 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:11:59.203808 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:59.203815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:59.203817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:59.204048 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:59.204259 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:11:59.204260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:11:59.204261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:11:59.204378 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:11:59.204704 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:11:59.204711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:11:59.204886 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 12:11:59.205024 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 12:11:59.205183 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260925 12:11:59.205950 6684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:11:59.206058 6684 heartbeater.cc:464] Registering TS with master...
I20260925 12:11:59.206247 6684 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:11:59.206830 6493 ts_manager.cc:194] Registered new tserver with Master: fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:11:59.207525 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 12:12:00.208467 6684 heartbeater.cc:503] Master 127.0.98.62:40957 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
W20260925 12:12:00.247172 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.2: real 1.051s user 0.000s sys 0.003s
I20260925 12:12:00.247619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:00.328938 6689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:00.329133 6689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:00.329155 6689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:00.330294 6689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:00.330335 6689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:00.330363 6689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:00.331775 6689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:00.332026 6689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:00.332265 6689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:00.337154 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.337186 6695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.337231 6689 server_base.cc:1034] running on GCE node
W20260925 12:12:00.337160 6697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.337621 6689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:00.337870 6689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:00.339020 6689 hybrid_clock.cc:648] HybridClock initialized: now 1790338320338998 us; error 38 us; skew 500 ppm
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:00.347573 6689 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:12:00.348042 6689 webserver.cc:460] Webserver started at http://127.0.98.2:46677/ using document root <none> and password file <none>
I20260925 12:12:00.348255 6689 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.348299 6689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.348395 6689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:00.349493 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "cf9884e2074344ea95cc53766d2ee9be"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "219fa0635d549b263997585cabefafbe"
server_key_iv: "29072c3b8286daebe462fd99bf7bcc79"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.349794 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cf9884e2074344ea95cc53766d2ee9be"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "219fa0635d549b263997585cabefafbe"
server_key_iv: "29072c3b8286daebe462fd99bf7bcc79"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.351063 6689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:00.351853 6704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.352001 6689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:00.352082 6689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "cf9884e2074344ea95cc53766d2ee9be"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "219fa0635d549b263997585cabefafbe"
server_key_iv: "29072c3b8286daebe462fd99bf7bcc79"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.352159 6689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.365724 6689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:00.367807 6689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:00.367964 6689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:00.368191 6689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:00.368546 6689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:00.368595 6689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.368650 6689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:00.368681 6689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.374212 6689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34509
I20260925 12:12:00.374259 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34509 every 8 connection(s)
I20260925 12:12:00.374565 6689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:00.382834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6689
I20260925 12:12:00.382937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:00.383153 392 external_mini_cluster.cc:1428] Setting key 0bb58a49777eb10c13bd727681c58594
I20260925 12:12:00.384779 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.375229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:43187)
0925 12:12:00.375549 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.375595 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.376121 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.376385 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.376389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.377025 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.377214 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.377219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.377954 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.377955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.378781 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.378783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.379205 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.379206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.379340 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.379785 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.379793 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.381345 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.383166 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.383171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.383172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.383462 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.383796 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.383796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.383797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.383946 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.384247 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.384251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.384364 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.384446 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.384621 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
I20260925 12:12:00.384783 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.375314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43187 (local address 127.0.98.62:40957)
0925 12:12:00.375540 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.375543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.375687 (+ 144us) server_negotiation.cc:407] Connection header received
0925 12:12:00.376233 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.376235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.376248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.376310 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.377325 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.377859 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.378877 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.379108 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.379195 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.381522 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.381523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.381524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.381531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.383089 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.383622 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.383623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.383623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.383691 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.384029 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.384030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.384031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.384179 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.384252 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.384511 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.384611 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":56,"threads_started":1}
I20260925 12:12:00.385311 6818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:12:00.385413 6818 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:00.385653 6818 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:12:00.386031 6493 ts_manager.cc:194] Registered new tserver with Master: cf9884e2074344ea95cc53766d2ee9be (127.0.98.2:34509)
I20260925 12:12:00.386409 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43187
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:12:00.407861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:00.489542 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:00.489742 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:00.489765 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:00.490882 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:00.490921 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:00.490945 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:00.492287 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:00.492461 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:00.492698 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:00.497717 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.497810 6824 server_base.cc:1034] running on GCE node
W20260925 12:12:00.497720 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.497720 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.498147 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:00.498445 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:00.499593 6824 hybrid_clock.cc:648] HybridClock initialized: now 1790338320499563 us; error 45 us; skew 500 ppm
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:00.502202 6824 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:12:00.502708 6824 webserver.cc:460] Webserver started at http://127.0.98.3:45325/ using document root <none> and password file <none>
I20260925 12:12:00.502982 6824 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.503041 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.503176 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:00.504258 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "7df81531e1b3455bbfd703ee9b62ecd6"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "e14037069800299d5d35a1050da9eab5"
server_key_iv: "e3a88dbefed955cba66e9bb45daf340d"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.504647 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7df81531e1b3455bbfd703ee9b62ecd6"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "e14037069800299d5d35a1050da9eab5"
server_key_iv: "e3a88dbefed955cba66e9bb45daf340d"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.506026 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:00.506763 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.506951 6824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:00.507036 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "7df81531e1b3455bbfd703ee9b62ecd6"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "e14037069800299d5d35a1050da9eab5"
server_key_iv: "e3a88dbefed955cba66e9bb45daf340d"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.507126 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.515133 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:00.517458 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:00.517606 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:00.517823 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:00.518118 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:00.518152 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.518174 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:00.518242 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.523823 6824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43165
I20260925 12:12:00.523898 6952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43165 every 8 connection(s)
I20260925 12:12:00.524199 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:00.532545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6824
I20260925 12:12:00.532675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:00.532891 392 external_mini_cluster.cc:1428] Setting key cb6a1d2cb22a03b7771f8b2f2783c09f
I20260925 12:12:00.533383 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.524962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35059 (local address 127.0.98.62:40957)
0925 12:12:00.525043 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.525046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.525271 (+ 225us) server_negotiation.cc:407] Connection header received
0925 12:12:00.525724 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.525725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.525734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.525788 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.526707 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.527207 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.528173 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.528380 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.528436 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.530552 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.530554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.530554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.530560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.531836 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.532205 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.532206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.532206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.532313 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.532612 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.532613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.532613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.532756 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.532824 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.533157 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.533270 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 12:12:00.533532 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.524937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:35059)
0925 12:12:00.525138 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.525173 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.525627 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.525859 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.525863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.526406 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.526595 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.526599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.527289 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.527290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.528084 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.528086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.528454 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.528455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.528627 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.529146 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.529154 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.530469 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.531910 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.531915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.531917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.532118 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.532384 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.532384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.532384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.532474 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.532829 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.532836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.533002 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.533126 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.533305 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260925 12:12:00.533933 6953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:12:00.534051 6953 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:00.534229 6953 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:12:00.534593 6493 ts_manager.cc:194] Registered new tserver with Master: 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165)
I20260925 12:12:00.534956 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35059
I20260925 12:12:00.544795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 12:12:00.571645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35463
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40957,127.0.98.61:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:00.654374 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:00.654603 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:00.654627 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:00.655921 6958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:00.655992 6958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:00.656020 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:00.656041 6958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:00.656073 6958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:00.657533 6958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40957,127.0.98.61:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
W20260925 12:12:00.657639 6958 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40957,127.0.98.61:35463'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 12:12:00.657754 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:00.657927 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:00.662796 6963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.662789 6964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.662789 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.662966 6958 server_base.cc:1034] running on GCE node
I20260925 12:12:00.663328 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:00.663591 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:00.664727 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790338320664711 us; error 42 us; skew 500 ppm
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:00.667181 6958 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 12:12:00.667680 6958 webserver.cc:460] Webserver started at http://127.0.98.61:40529/ using document root <none> and password file <none>
I20260925 12:12:00.667945 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.668030 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.668188 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:00.669361 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/instance:
uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3905d09bd578e263a608a5d3b963c416"
server_key_iv: "d3ac24cc4f8f914b24d1da0a2f49d431"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.669731 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal/instance:
uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3905d09bd578e263a608a5d3b963c416"
server_key_iv: "d3ac24cc4f8f914b24d1da0a2f49d431"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.671074 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:12:00.671797 6973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.671963 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:00.672062 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3905d09bd578e263a608a5d3b963c416"
server_key_iv: "d3ac24cc4f8f914b24d1da0a2f49d431"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.672147 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.692119 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:00.694564 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:00.694732 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:00.698920 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35463
I20260925 12:12:00.698957 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35463 every 8 connection(s)
I20260925 12:12:00.699314 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:12:00.700253 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:00.701845 7026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:12:00.706867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 6958
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:00.711262 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.702556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57356)
0925 12:12:00.702711 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.702755 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.703200 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.703418 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.703421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.703683 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.703877 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.703881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.704582 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.704583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.705410 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.705412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.705747 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.705747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.705848 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.706383 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.706391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.708151 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.709819 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.709825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.709827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.710065 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.710280 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.710281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.710281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.710391 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.710679 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.710686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.710846 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.710965 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.711098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260925 12:12:00.711604 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.702598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.98.62:40957)
0925 12:12:00.702662 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.702664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.702846 (+ 182us) server_negotiation.cc:407] Connection header received
0925 12:12:00.703288 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.703289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.703297 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.703359 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.703955 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.704442 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.705485 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.705673 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.705727 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.708243 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.708245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.708245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.708252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.709724 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.710141 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.710143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.710143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.710204 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.710455 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.710456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.710456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.710603 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.710677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.711435 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.711515 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:12:00.712168 7026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:12:00.715692 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.707402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.0.98.61:35463)
0925 12:12:00.707605 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.707610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.707636 (+ 26us) server_negotiation.cc:407] Connection header received
0925 12:12:00.707659 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.707661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.707681 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.707764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.708388 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.709126 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.709875 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.710167 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.710249 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.712950 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.712957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.712958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.712964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.714482 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.714855 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.714858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.714858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.714956 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.715157 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.715158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.715159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.715292 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.715467 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.715512 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.715586 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260925 12:12:00.718619 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.712427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33492)
0925 12:12:00.712467 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.712477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.712531 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.712765 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.712765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.712906 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.712948 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.712949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.713578 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.713578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.714042 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.714043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.714409 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.714409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.714471 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.714859 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.714864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.715993 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.717841 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.717842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.717842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.718007 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.718151 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.718151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.718152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.718217 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.718410 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.718411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.718448 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.718498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.718566 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 12:12:00.718770 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.712477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.98.61:35463)
0925 12:12:00.712662 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.712664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.712669 (+ 5us) server_negotiation.cc:407] Connection header received
0925 12:12:00.712682 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.712683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.712690 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.712763 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.713027 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.713495 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.714132 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.714339 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.714411 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.716065 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.716065 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.716066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.716071 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.717776 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.718062 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.718062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.718063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.718101 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.718273 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.718274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.718274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.718357 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.718398 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.718511 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.718605 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260925 12:12:00.719947 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Bootstrap starting.
I20260925 12:12:00.720634 7026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:00.720907 7026 log.cc:824] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:00.721629 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: No bootstrap required, opened a new log
I20260925 12:12:00.722970 7026 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } }
I20260925 12:12:00.723111 7026 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:00.723156 7026 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcba8cf5dc0c414ca1124072fcb6dbf7, State: Initialized, Role: FOLLOWER
I20260925 12:12:00.723304 7026 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } }
I20260925 12:12:00.723603 7026 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:00.723563 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } } }
I20260925 12:12:00.723702 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:12:00.725298 7044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:12:00.725363 7044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:12:00.726874 6958 master_runner.cc:428] Detected that this master fcba8cf5dc0c414ca1124072fcb6dbf7 is joining an existing cluster
I20260925 12:12:00.726912 6958 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35463
I20260925 12:12:00.734159 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.727829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.98.62:40957)
0925 12:12:00.727923 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.727925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.728160 (+ 235us) server_negotiation.cc:407] Connection header received
0925 12:12:00.728223 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.728224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.728234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.728282 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.728677 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.729160 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.729924 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.730118 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.730201 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.731185 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.731187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.731187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.731194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.732794 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.733236 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.733237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.733237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.733292 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.733494 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.733495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.733495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.733625 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.733693 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.733961 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.734043 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 12:12:00.734185 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.727988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57362)
0925 12:12:00.728108 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.728117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.728165 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.728283 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.728284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.728481 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.728594 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.728595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.729257 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.729258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.729841 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.729842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.730220 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.730221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.730288 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.730746 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.730750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.731082 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.732878 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.732879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.732880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.733149 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.733366 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.733367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.733367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.733422 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.733685 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.733685 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.733741 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.733821 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.733922 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260925 12:12:00.734854 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.727850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33504)
0925 12:12:00.728001 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.728011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.728071 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.728227 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.728228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.728456 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.728563 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.728565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.729478 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.729479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.730322 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.730324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.730811 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.730812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.730897 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.731291 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.731296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.731634 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.733597 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.733598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.733598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.733820 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.734007 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.734007 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.734008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.734054 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.734500 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.734501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.734560 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.734640 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.734738 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260925 12:12:00.734910 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.727867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.98.61:35463)
0925 12:12:00.727954 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.727956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.728060 (+ 104us) server_negotiation.cc:407] Connection header received
0925 12:12:00.728125 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.728126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.728136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.728191 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.728646 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.729383 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.730411 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.730713 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.730748 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.731722 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.731723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.731723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.731729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.733528 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.733898 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.733899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.733899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.733944 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.734115 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.734115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.734116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.734250 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.734293 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.734661 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.734796 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":70}
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:12:00.744628 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.737363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33512)
0925 12:12:00.737474 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.737519 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.737577 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.737715 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.737717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.737948 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.738007 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.738008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.738915 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.738915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.739487 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.739488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.739954 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.739955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.740025 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.740468 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.740473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.742001 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.743599 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.743605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.743607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.743792 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.743990 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.743991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.743991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.744043 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.744290 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.744296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.744338 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.744404 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.744546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260925 12:12:00.744632 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.737425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.0.98.61:35463)
0925 12:12:00.737466 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.737468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.737579 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:12:00.737632 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.737633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.737643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.737711 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.738078 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.738822 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.739565 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.739867 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.739936 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.742080 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.742082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.742082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.742089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.743519 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.743867 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.743868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.743869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.743925 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.744101 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.744102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.744102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.744227 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.744292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.744447 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.744538 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:12:00.745605 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35463: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35463
}
attrs {
promote: true
}
}
dest_uuid: "bba85b6019884f15a1119d6a183f1763"
cas_config_opid_index: -1
I20260925 12:12:00.745941 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } }
I20260925 12:12:00.746486 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fcba8cf5dc0c414ca1124072fcb6dbf7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } }
I20260925 12:12:00.746780 7000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:00.747584 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35463
I20260925 12:12:00.747718 7000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Refusing update from remote peer bba85b6019884f15a1119d6a183f1763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 12:12:00.747751 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } }
I20260925 12:12:00.747829 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:00.747967 6958 master.cc:537] Master@127.0.98.61:35463 shutting down...
I20260925 12:12:00.748055 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:12:00.748759 6477 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 12:12:00.749147 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 -> Peer fcba8cf5dc0c414ca1124072fcb6dbf7 (127.0.98.61:35463): Couldn't send request to peer fcba8cf5dc0c414ca1124072fcb6dbf7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 12:12:00.751030 6958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:00.751117 6958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:00.751168 6958 tablet_replica.cc:330] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: stopping tablet replica
I20260925 12:12:00.754693 6958 master.cc:552] Master@127.0.98.61:35463 shutdown complete.
I20260925 12:12:00.754755 6958 master_runner.cc:305] Clearing existing system tablet
I20260925 12:12:00.754891 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.754948 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.755534 6958 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:00.756013 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.756161 6958 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.756233 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.756263 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3905d09bd578e263a608a5d3b963c416"
server_key_iv: "d3ac24cc4f8f914b24d1da0a2f49d431"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.756919 6958 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:00.758221 6958 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 12:12:00.758304 6958 log.cc:1196] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 12:12:00.758495 6958 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Deleting consensus metadata
I20260925 12:12:00.758630 6958 master_runner.cc:315] Copying system tablet from 127.0.98.62:40957
I20260925 12:12:00.759168 6958 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40957
I20260925 12:12:00.765110 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.759510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.98.62:40957)
0925 12:12:00.759557 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.759559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.759637 (+ 78us) server_negotiation.cc:407] Connection header received
0925 12:12:00.759712 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.759714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.759724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.759773 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.760066 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.760646 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.761361 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.761561 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.761640 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.762583 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.762584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.762584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.762591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.763984 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.764342 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.764343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.764344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.764401 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.764616 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.764617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.764617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.764727 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.764803 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.764951 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.765023 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:12:00.765228 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.759445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57368)
0925 12:12:00.759558 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.759570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.759653 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.759775 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.759776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.759944 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.759990 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.759992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.760720 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.760722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.761268 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.761270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.761642 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.761643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.761711 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.762200 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.762204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.762479 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.764082 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.764083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.764083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.764257 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.764464 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.764465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.764466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.764543 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.764794 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.764794 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.764838 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.764926 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.765023 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":73,"threads_started":1}
I20260925 12:12:00.765455 6513 tablet_copy_service.cc:140] P bba85b6019884f15a1119d6a183f1763: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fcba8cf5dc0c414ca1124072fcb6dbf7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57368)
I20260925 12:12:00.765533 6513 tablet_copy_service.cc:161] P bba85b6019884f15a1119d6a183f1763: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fcba8cf5dc0c414ca1124072fcb6dbf7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57368: session id = fcba8cf5dc0c414ca1124072fcb6dbf7-00000000000000000000000000000000
I20260925 12:12:00.766131 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 12:12:00.766769 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 12:12:00.768046 6958 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: tablet copy: Starting download of 0 data blocks...
I20260925 12:12:00.768178 6958 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: tablet copy: Starting download of 1 WAL segments...
I20260925 12:12:00.768978 6958 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 12:12:00.769819 6513 tablet_copy_service.cc:342] P bba85b6019884f15a1119d6a183f1763: Request end of tablet copy session fcba8cf5dc0c414ca1124072fcb6dbf7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57368
I20260925 12:12:00.769891 6513 tablet_copy_service.cc:433] P bba85b6019884f15a1119d6a183f1763: ending tablet copy session fcba8cf5dc0c414ca1124072fcb6dbf7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fcba8cf5dc0c414ca1124072fcb6dbf7
I20260925 12:12:00.772873 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:00.774067 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.774071 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.774310 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.776647 6958 server_base.cc:1034] running on GCE node
I20260925 12:12:00.776746 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:00.776895 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:00.778031 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790338320778031 us; error 32 us; skew 500 ppm
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:00.779301 6958 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 12:12:00.779573 6958 webserver.cc:460] Webserver started at http://127.0.98.61:46363/ using document root <none> and password file <none>
I20260925 12:12:00.779695 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.779754 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.780448 6958 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:00.781015 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.781162 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.781215 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3905d09bd578e263a608a5d3b963c416"
server_key_iv: "d3ac24cc4f8f914b24d1da0a2f49d431"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.781272 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.789503 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:00.791136 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:00.794188 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35463
I20260925 12:12:00.794389 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35463 every 8 connection(s)
I20260925 12:12:00.794605 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:12:00.795360 7129 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:12:00.795744 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Bootstrap starting.
I20260925 12:12:00.797899 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:12:00.798106 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: Bootstrap complete.
I20260925 12:12:00.798311 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } }
I20260925 12:12:00.798379 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fcba8cf5dc0c414ca1124072fcb6dbf7, State: Initialized, Role: LEARNER
I20260925 12:12:00.798405 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } }
I20260925 12:12:00.798571 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:00.798563 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } }
I20260925 12:12:00.798671 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:12:00.799896 7143 catalog_manager.cc:1259] Loaded cluster ID: 1a8051be89a64c51810eaa51b3efcbb6
I20260925 12:12:00.799940 7143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: loading cluster ID for follower catalog manager: success
I20260925 12:12:00.801065 7143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: acquiring CA information for follower catalog manager: success
I20260925 12:12:00.801342 7143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:12:00.824649 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.817278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.0.98.61:35463)
0925 12:12:00.817407 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.817408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.817415 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:00.817437 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.817438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.817446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.817496 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.818270 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.818915 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.819983 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.820182 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.820226 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.821745 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.821746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.821746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.821752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.823207 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.823669 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.823670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.823670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.823728 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.824016 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.824017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.824017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.824123 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.824200 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.824367 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.824427 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 12:12:00.851358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44431
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:00.934262 7147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:00.934450 7147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:00.934471 7147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:00.935570 7147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:00.935633 7147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:00.935662 7147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:00.935681 7147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:00.935703 7147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:00.937215 7147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:00.937402 7147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:00.937561 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:12:00.942348 7147 server_base.cc:1034] running on GCE node
W20260925 12:12:00.942353 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.942332 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:00.942323 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:00.942785 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:00.943063 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:00.944339 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790338320944321 us; error 28 us; skew 500 ppm
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:00.946887 7147 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 12:12:00.947333 7147 webserver.cc:460] Webserver started at http://127.0.98.60:39581/ using document root <none> and password file <none>
I20260925 12:12:00.947554 7147 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:00.947621 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:00.947774 7147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:00.948887 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/instance:
uuid: "0b413adb1ec4460d94fa3811ef4d924e"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3d879c34fc076d3edde055b64faffa15"
server_key_iv: "476d1b6570ef809d83bc776477aa4752"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.949236 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal/instance:
uuid: "0b413adb1ec4460d94fa3811ef4d924e"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3d879c34fc076d3edde055b64faffa15"
server_key_iv: "476d1b6570ef809d83bc776477aa4752"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.950538 7147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:00.951267 7162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:00.951428 7147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:00.951534 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "0b413adb1ec4460d94fa3811ef4d924e"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3d879c34fc076d3edde055b64faffa15"
server_key_iv: "476d1b6570ef809d83bc776477aa4752"
server_key_version: "encryptionkey@0"
I20260925 12:12:00.951607 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:00.965202 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:00.967003 7147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:00.967162 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:00.971882 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44431
I20260925 12:12:00.971925 7214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44431 every 8 connection(s)
I20260925 12:12:00.972261 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:12:00.973321 7215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:00.974841 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:12:00.976426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 7147
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:12:00.984866 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.975540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.98.62:40957)
0925 12:12:00.975615 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.975618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.975745 (+ 127us) server_negotiation.cc:407] Connection header received
0925 12:12:00.976247 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.976249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.976258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.976316 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.977205 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.977738 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.978749 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.978935 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.978984 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.981758 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.981760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.981761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.981769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.983344 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.983829 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.983830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.983831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.983881 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.984137 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.984137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.984138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.984253 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.984308 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.984649 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.984740 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:12:00.984973 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.975422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57378)
0925 12:12:00.975604 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.975649 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.976126 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.976419 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.976423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.976827 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.977110 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.977116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.977882 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.977882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.978634 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.978636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.979015 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.979016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.979150 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.979747 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.979756 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.981614 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.983417 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.983422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.983424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.983756 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.983949 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.983950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.983950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.984070 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.984327 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.984331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.984507 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.984623 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.984730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 12:12:00.985539 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:12:00.985803 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.976947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.98.60:44431)
0925 12:12:00.977103 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.977113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.977129 (+ 16us) server_negotiation.cc:407] Connection header received
0925 12:12:00.977145 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.977147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.977178 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.977255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.977815 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.978618 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.979525 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.979906 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.979961 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.982674 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.982680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.982681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.982687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.984435 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.984842 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.984845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.984846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.984901 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.985137 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.985138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.985138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.985283 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.985459 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.985524 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.985593 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 12:12:00.991891 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.985786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33542)
0925 12:12:00.985832 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.985843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.985879 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.986283 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.986284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.986418 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.986469 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.986471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.987111 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.987112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.987596 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.987597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.987981 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.987982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.988058 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.988454 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.988459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.989517 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.990918 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.990919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.990919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.991079 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.991279 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.991280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.991281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.991334 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.991578 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.991579 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.991619 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.991705 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.991800 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:12:00.991891 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.985871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.0.98.61:35463)
0925 12:12:00.985916 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.985918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.985925 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:00.985950 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.985951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.985961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.986017 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.986546 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.987024 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.987684 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.987900 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.987980 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.989601 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.989603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.989603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.989610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.990824 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.991149 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.991150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.991151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.991209 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.991404 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.991405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.991405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.991514 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.991586 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.991731 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.991806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:12:00.992549 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:12:00 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338320, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:12:00.998355 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.992774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.0.1:52336)
0925 12:12:00.992825 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:00.992835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:00.992945 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:00.993246 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:00.993247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:00.993428 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:00.993455 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.993457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.994032 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.994033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.994483 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:00.994485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.994827 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.994828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:00.994865 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.995239 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:00.995250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:00.996315 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:00.997610 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.997611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.997611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.997759 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.997925 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:00.997926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:00.997926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:00.997973 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:00.998137 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:00.998137 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:00.998178 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 12:12:00.998227 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:12:00.998299 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:12:00.998525 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:00.993074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.0.98.60:44431)
0925 12:12:00.993146 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:00.993147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:00.993154 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:00.993170 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:00.993170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:00.993178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:00.993230 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:00.993520 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.993959 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.994552 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:00.994733 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:00.994811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:00.996387 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:00.996388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:00.996389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:00.996394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:00.997542 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.997822 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.997823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.997823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.997867 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:00.998011 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:00.998012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:00.998012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:00.998089 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:00.998131 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:00.998244 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:12:00.998385 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":55}
I20260925 12:12:01.000005 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Bootstrap starting.
I20260925 12:12:01.001344 7215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:01.001617 7215 log.cc:824] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:01.002321 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: No bootstrap required, opened a new log
I20260925 12:12:01.003697 7215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44431 } }
I20260925 12:12:01.003859 7215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:01.003908 7215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b413adb1ec4460d94fa3811ef4d924e, State: Initialized, Role: FOLLOWER
I20260925 12:12:01.004041 7215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44431 } }
I20260925 12:12:01.004354 7215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:01.004338 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44431 } } }
I20260925 12:12:01.004452 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:12:01.006102 7231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:12:01.006166 7231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:12:01.008580 7147 master_runner.cc:428] Detected that this master 0b413adb1ec4460d94fa3811ef4d924e is joining an existing cluster
I20260925 12:12:01.008630 7147 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44431
I20260925 12:12:01.016880 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.009783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.98.61:35463)
0925 12:12:01.009830 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.009832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.009959 (+ 127us) server_negotiation.cc:407] Connection header received
0925 12:12:01.010003 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.010004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.010016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.010068 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.010529 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011355 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012470 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.012772 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012819 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.013873 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.013875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.013875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.013882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.015417 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.016098 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.016099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.016099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.016162 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.016338 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.016338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.016339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.016470 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.016559 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.016708 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.016790 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:12:01.016995 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.009700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33558)
0925 12:12:01.009868 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.009879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.009944 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.010120 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.010121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.010368 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.010441 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.010442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011467 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.011467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.012379 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.012381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.012888 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.012943 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.013447 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.013452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.013771 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.015507 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.015508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.015508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.015664 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.016229 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.016229 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.016230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.016276 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.016559 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.016559 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.016612 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.016662 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.016734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":32,"thread_start_us":71,"threads_started":1}
I20260925 12:12:01.036761 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.010028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.0.1:52342)
0925 12:12:01.010143 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.010152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.010223 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.010403 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.010404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.010649 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.010708 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.010710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011517 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.011517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.012379 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.012381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.012886 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.012942 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.033169 (+ 20227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.033180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.033701 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.035630 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.035631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.035632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.035846 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.036047 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.036048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.036049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.036110 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.036371 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.036372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.036420 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.036523 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.036640 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":34,"thread_start_us":57,"threads_started":1}
I20260925 12:12:01.037003 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.009733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.0.98.62:40957)
0925 12:12:01.009793 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.009795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.010193 (+ 398us) server_negotiation.cc:407] Connection header received
0925 12:12:01.010308 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.010309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.010319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.010350 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.010849 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011597 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012705 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.012992 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.013032 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.034031 (+ 20999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.034033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.034033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.034043 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.035690 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.036104 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.036105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.036106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.036160 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.036362 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.036363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.036364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.036539 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.036586 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.036802 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.036894 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:12:01.037051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6555
I20260925 12:12:01.037103 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.010057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.60:44431)
0925 12:12:01.010091 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.010093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.010195 (+ 102us) server_negotiation.cc:407] Connection header received
0925 12:12:01.010313 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.010314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.010323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.010351 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.010781 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011440 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012469 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.012762 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.012802 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.033837 (+ 21035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.033841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.033841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.033853 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.035534 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.035922 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.035923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.035924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.035974 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.036165 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.036166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.036166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.036304 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.036371 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.036551 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.037018 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":384,"server-negotiator.queue_time_us":19}
I20260925 12:12:01.038204 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.009856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57380)
0925 12:12:01.010139 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.010149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.010225 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.010413 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.010413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.010685 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.010755 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.010757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.011695 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.011696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.012607 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.012610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.013110 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.013111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.013174 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.033128 (+ 19954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.033135 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.033715 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.035793 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.035794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.035795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.036015 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.036231 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.036232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.036232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.036277 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.036635 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.036635 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.036687 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.038040 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.038123 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":65,"threads_started":1}
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:12:01.044797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40957
--webserver_port=34223
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:12:01.048013 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.040141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.0.1:52352)
0925 12:12:01.040191 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.040202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.040246 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.040381 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.040381 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.040578 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.040629 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.040631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.041601 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.041602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.042536 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.042539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.043061 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.043062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.043133 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.043534 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.043539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.045084 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.046862 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.046863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.046864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.047123 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.047343 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.047344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.047344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.047409 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.047679 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.047679 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.047726 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.047819 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.047918 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:12:01.048012 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.040169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.0.98.60:44431)
0925 12:12:01.040201 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.040204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.040232 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:12:01.040323 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.040324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.040332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.040380 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.040711 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.041509 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.042626 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.042959 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.043015 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.045155 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.045156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.045156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.045163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.046767 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.047200 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.047202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.047202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.047267 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.047472 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.047472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.047473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.047611 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.047683 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.047842 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.047918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:12:01.049008 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44431: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44431
}
attrs {
promote: true
}
}
dest_uuid: "bba85b6019884f15a1119d6a183f1763"
cas_config_opid_index: 5
I20260925 12:12:01.049265 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:01.049587 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b413adb1ec4460d94fa3811ef4d924e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.050122 7189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:01.050571 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44431
I20260925 12:12:01.050639 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.050697 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:01.050971 7147 master.cc:537] Master@127.0.98.60:44431 shutting down...
I20260925 12:12:01.051115 7189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 1 FOLLOWER]: Refusing update from remote peer bba85b6019884f15a1119d6a183f1763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 12:12:01.051354 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:12:01.052004 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 -> Peer 0b413adb1ec4460d94fa3811ef4d924e (127.0.98.60:44431): Couldn't send request to peer 0b413adb1ec4460d94fa3811ef4d924e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 12:12:01.054163 7147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:01.054265 7147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:01.054314 7147 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: stopping tablet replica
I20260925 12:12:01.056314 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.049695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.0.1:33572)
0925 12:12:01.052238 (+ 2543us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.052250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.052302 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.052612 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.052613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.052614 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.052656 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.052657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.053586 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.053587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.054745 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.054747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.056018 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.056019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.056092 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.056118 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.056168 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.056213 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2525}
I20260925 12:12:01.056586 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.049799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.98.61:35463)
0925 12:12:01.049843 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.049844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.052360 (+ 2516us) server_negotiation.cc:407] Connection header received
0925 12:12:01.052389 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.052391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.052416 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.052472 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.052734 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.053494 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.054848 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.055477 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.055563 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.055652 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.056465 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.056530 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:12:01.057006 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 LEARNER]: Refusing update from remote peer bba85b6019884f15a1119d6a183f1763: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 12:12:01.057266 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 12:12:01.057924 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b413adb1ec4460d94fa3811ef4d924e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.059101 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba85b6019884f15a1119d6a183f1763. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.059111 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: true } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.059208 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:12:01.059216 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: This master's current role is: LEARNER
I20260925 12:12:01.059540 7054 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: attempting to promote NON_VOTER fcba8cf5dc0c414ca1124072fcb6dbf7 to VOTER
I20260925 12:12:01.059721 7054 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:01.060153 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 LEARNER]: Refusing update from remote peer bba85b6019884f15a1119d6a183f1763: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 12:12:01.060195 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.060102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.0.1:52360)
0925 12:12:01.060135 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.060168 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 12:12:01.060360 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 12:12:01.060474 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.060404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.0.1:52370)
0925 12:12:01.060432 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.060453 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 12:12:01.060606 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 -> Peer 0b413adb1ec4460d94fa3811ef4d924e (127.0.98.60:44431): Couldn't send request to peer 0b413adb1ec4460d94fa3811ef4d924e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 12:12:01.061035 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fcba8cf5dc0c414ca1124072fcb6dbf7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.061161 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fcba8cf5dc0c414ca1124072fcb6dbf7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.061971 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.062052 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:01.062109 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.062172 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:01.062263 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bba85b6019884f15a1119d6a183f1763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.062359 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:12:01.070093 7147 master.cc:552] Master@127.0.98.60:44431 shutdown complete.
I20260925 12:12:01.070150 7147 master_runner.cc:305] Clearing existing system tablet
I20260925 12:12:01.070286 7147 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:01.070334 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:01.071108 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:01.071624 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.071780 7147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:01.071841 7147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.071867 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "0b413adb1ec4460d94fa3811ef4d924e"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3d879c34fc076d3edde055b64faffa15"
server_key_iv: "476d1b6570ef809d83bc776477aa4752"
server_key_version: "encryptionkey@0"
I20260925 12:12:01.072584 7147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:01.074122 7147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 12:12:01.074191 7147 log.cc:1196] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 12:12:01.074375 7147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Deleting consensus metadata
I20260925 12:12:01.074488 7147 master_runner.cc:315] Copying system tablet from 127.0.98.62:40957
I20260925 12:12:01.075079 7147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40957
I20260925 12:12:01.082791 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.075452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.0.98.62:40957)
0925 12:12:01.075501 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.075503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.075554 (+ 51us) server_negotiation.cc:407] Connection header received
0925 12:12:01.075621 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.075622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.075633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.075706 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.076040 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.076871 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.077937 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.078237 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.078310 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.079857 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.079859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.079859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.079866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.081563 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.081964 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.081965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.081966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.082026 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.082229 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.082230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.082230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.082364 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.082430 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.082597 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.082669 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:12:01.082798 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.075381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:57390)
0925 12:12:01.075497 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.075508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.075568 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.075709 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.075710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.075904 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.075961 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.075963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.076957 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.076958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.077838 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.077841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.078321 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.078322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.078386 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.079344 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.079349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.079770 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.081673 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.081675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.081675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.081891 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.082100 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.082101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.082101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.082164 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.082442 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.082443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.082488 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.082559 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.082665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260925 12:12:01.083444 6513 tablet_copy_service.cc:140] P bba85b6019884f15a1119d6a183f1763: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b413adb1ec4460d94fa3811ef4d924e ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57390)
I20260925 12:12:01.083513 6513 tablet_copy_service.cc:161] P bba85b6019884f15a1119d6a183f1763: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b413adb1ec4460d94fa3811ef4d924e at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57390: session id = 0b413adb1ec4460d94fa3811ef4d924e-00000000000000000000000000000000
I20260925 12:12:01.084087 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 12:12:01.084635 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 12:12:01.085889 7147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: tablet copy: Starting download of 0 data blocks...
I20260925 12:12:01.086027 7147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: tablet copy: Starting download of 1 WAL segments...
I20260925 12:12:01.086810 7147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 12:12:01.087730 6513 tablet_copy_service.cc:342] P bba85b6019884f15a1119d6a183f1763: Request end of tablet copy session 0b413adb1ec4460d94fa3811ef4d924e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57390
I20260925 12:12:01.087783 6513 tablet_copy_service.cc:433] P bba85b6019884f15a1119d6a183f1763: ending tablet copy session 0b413adb1ec4460d94fa3811ef4d924e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b413adb1ec4460d94fa3811ef4d924e
I20260925 12:12:01.090193 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:01.091411 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.091560 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.092664 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:01.092661 7147 server_base.cc:1034] running on GCE node
I20260925 12:12:01.092806 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:01.092940 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:01.094079 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790338321094069 us; error 23 us; skew 500 ppm
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:01.095531 7147 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 12:12:01.095774 7147 webserver.cc:460] Webserver started at http://127.0.98.60:40689/ using document root <none> and password file <none>
I20260925 12:12:01.095890 7147 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:01.095924 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:01.096678 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:01.097271 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.097430 7147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:01.097481 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "0b413adb1ec4460d94fa3811ef4d924e"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "3d879c34fc076d3edde055b64faffa15"
server_key_iv: "476d1b6570ef809d83bc776477aa4752"
server_key_version: "encryptionkey@0"
I20260925 12:12:01.097533 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:01.106644 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:01.108637 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:01.111706 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44431
I20260925 12:12:01.111797 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44431 every 8 connection(s)
I20260925 12:12:01.112197 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:12:01.113180 7316 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:12:01.113686 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Bootstrap starting.
I20260925 12:12:01.116577 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:12:01.116813 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: Bootstrap complete.
I20260925 12:12:01.117013 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:01.117102 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b413adb1ec4460d94fa3811ef4d924e, State: Initialized, Role: LEARNER
I20260925 12:12:01.117136 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:01.117288 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:01.118957 7330 catalog_manager.cc:1259] Loaded cluster ID: 1a8051be89a64c51810eaa51b3efcbb6
I20260925 12:12:01.119170 7330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: loading cluster ID for follower catalog manager: success
I20260925 12:12:01.120339 7330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: acquiring CA information for follower catalog manager: success
I20260925 12:12:01.120736 7330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:12:01.124625 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } } }
I20260925 12:12:01.124739 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b413adb1ec4460d94fa3811ef4d924e [sys.catalog]: This master's current role is: LEARNER
W20260925 12:12:01.148695 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:01.148931 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:01.148970 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:01.150130 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:01.150199 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:01.150246 7240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:01.151782 7240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34223
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:01.152051 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:01.152297 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:01.158571 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.158555 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.158547 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:01.159147 7240 server_base.cc:1034] running on GCE node
I20260925 12:12:01.159374 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:01.159647 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:01.160825 7240 hybrid_clock.cc:648] HybridClock initialized: now 1790338321160805 us; error 27 us; skew 500 ppm
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:01.163848 7240 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:12:01.164366 7240 webserver.cc:460] Webserver started at http://127.0.98.1:34223/ using document root <none> and password file <none>
I20260925 12:12:01.164649 7240 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:01.164731 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:01.166136 7240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:01.166790 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.166976 7240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:01.167076 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "fc345b8b302144dab0ff926d26851f2f"
format_stamp: "Formatted at 2026-09-25 12:11:59 on dist-test-slave-2hqq"
server_key: "e9baad2a85e7c1be1eb3e35c8460062c"
server_key_iv: "cb4f05178f0d25f24508930197998012"
server_key_version: "encryptionkey@0"
I20260925 12:12:01.167471 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:01.187844 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:01.190382 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:01.190554 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:01.190805 7240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:01.191157 7240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:01.191211 7240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.191252 7240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:01.191286 7240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.197726 7240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46681
I20260925 12:12:01.197767 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46681 every 8 connection(s)
I20260925 12:12:01.198154 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:01.201351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 7240
I20260925 12:12:01.201499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6689
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:01.208833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34509
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40957
--webserver_port=46677
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:12:01.210850 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.198881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60747 (local address 127.0.98.60:44431)
0925 12:12:01.199059 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.199061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.199390 (+ 329us) server_negotiation.cc:407] Connection header received
0925 12:12:01.199970 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.199971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.199983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.200040 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.200944 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.201909 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203433 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.203646 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.207529 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.207530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.207531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.207539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.209267 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.209698 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.209699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.209700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.209770 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.210046 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.210048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.210048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.210170 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.210244 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.210624 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.210723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
I20260925 12:12:01.211639 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.198893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.98.1:60747)
0925 12:12:01.199248 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.199258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.199808 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.200094 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.200095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.200579 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.200838 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.200843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.202021 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.202022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.203339 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.203341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.203745 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.203881 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.204542 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.204552 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.207438 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.209368 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.209377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.209379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.209608 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.209841 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.209842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.209842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.209971 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.210256 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.210264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.210457 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.211184 (+ 727us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.211402 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":59,"spinlock_wait_cycles":7552,"thread_start_us":93,"threads_started":1}
I20260925 12:12:01.212064 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.198925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36611 (local address 127.0.98.61:35463)
0925 12:12:01.198992 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.198994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.199240 (+ 246us) server_negotiation.cc:407] Connection header received
0925 12:12:01.199931 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.199933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.199945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.199976 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.200947 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.201705 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203438 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.203794 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203873 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.208261 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.208262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.208263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.208271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.210476 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.211199 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.211200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.211201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.211257 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.211462 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.211463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.211464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.211602 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.211645 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.211864 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.211943 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:12:01.212405 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44431
I20260925 12:12:01.212548 7459 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.212738 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.198893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.98.1:36611)
0925 12:12:01.199060 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.199107 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.199807 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.200071 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.200076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.200574 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.200838 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.200843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.202177 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.202177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.203339 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.203341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.203870 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.203871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.203924 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.204644 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.204648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.208169 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.210575 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.210576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.210576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.211105 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.211333 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.211334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.211334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.211391 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.211696 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.211696 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.211741 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.212183 (+ 442us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.212300 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260925 12:12:01.212875 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.199295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60953 (local address 127.0.98.62:40957)
0925 12:12:01.199388 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.199390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.199399 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:01.199910 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.199912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.199926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.199988 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.200940 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.201639 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.205517 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.205841 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.205907 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.208271 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.208273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.208273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.208281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.211568 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.211961 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.211963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.211964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.212025 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.212238 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.212238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.212239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.212364 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.212411 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.212663 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.212761 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 12:12:01.213238 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35463
I20260925 12:12:01.213325 7461 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.212747 7459 heartbeater.cc:511] Master 127.0.98.60:44431 requested a full tablet report, sending...
I20260925 12:12:01.213404 7461 heartbeater.cc:511] Master 127.0.98.61:35463 requested a full tablet report, sending...
I20260925 12:12:01.213589 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.199193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:60953)
0925 12:12:01.199330 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.199338 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.199816 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.200437 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.200438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.200655 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.200837 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.200841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.204411 (+ 3570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.204412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.205428 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.205431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.205963 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.205963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.206029 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.206455 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.206461 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.208159 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.211648 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.211649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.211649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.211866 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.212099 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.212099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.212099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.212166 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.212459 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.212460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.212542 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.213051 (+ 509us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.213169 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260925 12:12:01.213886 7093 ts_manager.cc:194] Registered new tserver with Master: fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:12:01.213896 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:12:01.213913 7276 ts_manager.cc:194] Registered new tserver with Master: fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:12:01.213939 7460 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.213997 7460 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:12:01.214295 6493 ts_manager.cc:194] Re-registered known tserver with Master: fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:12:01.214855 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60953
W20260925 12:12:01.293838 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:01.294037 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:01.294060 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:01.295130 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:01.295166 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:01.295192 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:01.296716 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46677
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:01.296896 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:01.297062 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:01.302629 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.302659 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.302644 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:01.302843 7467 server_base.cc:1034] running on GCE node
I20260925 12:12:01.303035 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:01.303224 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:01.304370 7467 hybrid_clock.cc:648] HybridClock initialized: now 1790338321304357 us; error 27 us; skew 500 ppm
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:01.306721 7467 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:12:01.307126 7467 webserver.cc:460] Webserver started at http://127.0.98.2:46677/ using document root <none> and password file <none>
I20260925 12:12:01.307401 7467 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:01.307489 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:01.308917 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:01.309559 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.309729 7467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:01.309836 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "cf9884e2074344ea95cc53766d2ee9be"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "219fa0635d549b263997585cabefafbe"
server_key_iv: "29072c3b8286daebe462fd99bf7bcc79"
server_key_version: "encryptionkey@0"
I20260925 12:12:01.310216 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:01.317346 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:01.319262 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:01.319412 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:01.319618 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:01.319913 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:01.319965 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.320004 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:01.320031 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.326747 7467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34509
I20260925 12:12:01.326814 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34509 every 8 connection(s)
I20260925 12:12:01.327158 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:12:01.335057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 7467
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:01.335213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6824
I20260925 12:12:01.339387 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.327919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41947 (local address 127.0.98.60:44431)
0925 12:12:01.327974 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.327976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.328404 (+ 428us) server_negotiation.cc:407] Connection header received
0925 12:12:01.329546 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.329548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.329563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.329595 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.330415 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.331168 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332314 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332625 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332667 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.335949 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.335950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.335951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.335959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.338009 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338392 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.338393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.338394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.338441 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338635 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.338636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.338636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.338760 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.338799 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.339184 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.339304 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:12:01.339381 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.327895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58883 (local address 127.0.98.61:35463)
0925 12:12:01.327986 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.327988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.328405 (+ 417us) server_negotiation.cc:407] Connection header received
0925 12:12:01.329549 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.329550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.329562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.329619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.330404 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.330902 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.331919 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332127 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332211 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.335989 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.335990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.335990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.335997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.337736 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338265 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.338266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.338266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.338321 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338599 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.338600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.338601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.338729 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.338775 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.339184 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.339285 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 12:12:01.339936 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.328594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40735 (local address 127.0.98.62:40957)
0925 12:12:01.328637 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.328639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.328647 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:01.329674 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.329676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.329687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.329745 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.330417 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.331176 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332337 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332661 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332750 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.335975 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.335976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.335977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.335984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.338660 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.339054 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.339056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.339056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.339109 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.339308 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.339309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.339310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.339449 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.339490 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.339705 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.339794 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:12:01.339943 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.327900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.98.2:58883)
0925 12:12:01.328285 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.328325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.329433 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.329697 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.329698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.330088 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.330304 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.330309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.331005 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.331006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.331817 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.331819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332219 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.332381 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.332983 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.332992 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.335604 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.337830 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.337838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.337840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.338167 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.338394 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.338395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.338523 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.338825 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.338832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.338993 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.339520 (+ 527us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.339714 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"thread_start_us":249,"threads_started":1}
I20260925 12:12:01.340310 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.328400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:40735)
0925 12:12:01.328554 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.328563 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.329437 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.329769 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.329769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.330104 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.330305 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.330311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.331271 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.331272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.332208 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.332209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332757 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.332826 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.333357 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.333362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.335872 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.338748 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.338749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.338977 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.339173 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.339174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.339174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.339228 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.339536 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.339537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.339598 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.340029 (+ 431us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.340144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":57,"thread_start_us":76,"threads_started":1}
I20260925 12:12:01.340690 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:12:01.340782 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35463
I20260925 12:12:01.340812 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.327902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.98.2:41947)
0925 12:12:01.328298 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.328324 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.329428 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.329686 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.329689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.330090 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.330305 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.330311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.331277 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.331278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.332208 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.332209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.332742 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.332743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.332814 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.333193 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.333197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.335868 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.338094 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.338095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.338309 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.338508 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.338508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.338509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.338563 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.338948 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.338949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.338993 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.339830 (+ 837us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.339913 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":54,"thread_start_us":204,"threads_started":1}
I20260925 12:12:01.340842 7597 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.341065 7597 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:12:01.341130 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44431
I20260925 12:12:01.341168 7596 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.341228 7596 heartbeater.cc:511] Master 127.0.98.60:44431 requested a full tablet report, sending...
I20260925 12:12:01.341476 6493 ts_manager.cc:194] Re-registered known tserver with Master: cf9884e2074344ea95cc53766d2ee9be (127.0.98.2:34509)
I20260925 12:12:01.341559 7276 ts_manager.cc:194] Registered new tserver with Master: cf9884e2074344ea95cc53766d2ee9be (127.0.98.2:34509)
I20260925 12:12:01.342000 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40735
I20260925 12:12:01.340842 7598 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.342420 7598 heartbeater.cc:511] Master 127.0.98.61:35463 requested a full tablet report, sending...
I20260925 12:12:01.342585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43165
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40957
--webserver_port=45325
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:12:01.342836 7093 ts_manager.cc:194] Registered new tserver with Master: cf9884e2074344ea95cc53766d2ee9be (127.0.98.2:34509)
W20260925 12:12:01.423730 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:01.423974 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:01.424010 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:01.425153 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:01.425218 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:01.425264 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:01.426797 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45325
--tserver_master_addrs=127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:01.427050 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:01.427289 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:01.432206 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.432224 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:01.432240 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:01.432425 7603 server_base.cc:1034] running on GCE node
I20260925 12:12:01.432669 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:01.432909 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:01.434067 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790338321434071 us; error 44 us; skew 500 ppm
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:01.436407 7603 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:12:01.436900 7603 webserver.cc:460] Webserver started at http://127.0.98.3:45325/ using document root <none> and password file <none>
I20260925 12:12:01.437119 7603 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:01.437201 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:01.438541 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:01.439090 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.439265 7603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.439366 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "7df81531e1b3455bbfd703ee9b62ecd6"
format_stamp: "Formatted at 2026-09-25 12:12:00 on dist-test-slave-2hqq"
server_key: "e14037069800299d5d35a1050da9eab5"
server_key_iv: "e3a88dbefed955cba66e9bb45daf340d"
server_key_version: "encryptionkey@0"
I20260925 12:12:01.439702 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:01.447204 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:01.449180 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:01.449333 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:01.449541 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:01.449827 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:01.449879 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.449919 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:01.449947 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:01.455554 7603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43165
I20260925 12:12:01.455595 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43165 every 8 connection(s)
I20260925 12:12:01.455942 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:12:01.457050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 7603
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 12:12:01 dist-test-slave-2hqq krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338321, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 12:12:01.468158 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.456666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35463 (local address 127.0.98.3:45629)
0925 12:12:01.457091 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.457102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.457586 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.457913 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.457915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.458373 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.458682 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.458688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459442 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.459443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.460219 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.460221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.460738 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.460740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.460881 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.461434 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.461445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.464509 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.466411 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.466417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.466419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.466724 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.466958 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.466959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.466959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.467099 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.467406 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.467411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.467581 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.467698 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.467896 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":30,"thread_start_us":92,"threads_started":1}
I20260925 12:12:01.468561 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.456645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45629 (local address 127.0.98.61:35463)
0925 12:12:01.456705 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.456707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.457189 (+ 482us) server_negotiation.cc:407] Connection header received
0925 12:12:01.457743 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.457745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.457760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.457813 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.458789 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459336 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.460312 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.460605 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.460672 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.464618 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.464619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.464619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.464627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.466320 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.466819 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.466820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.466821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.466882 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.467173 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.467174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.467174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.467315 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.467378 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.468356 (+ 978us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.468437 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:12:01.469276 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35463
I20260925 12:12:01.469386 7734 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.469592 7734 heartbeater.cc:511] Master 127.0.98.61:35463 requested a full tablet report, sending...
I20260925 12:12:01.470033 7093 ts_manager.cc:194] Registered new tserver with Master: 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165)
I20260925 12:12:01.471370 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.458308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.98.61:35463)
0925 12:12:01.460887 (+ 2579us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.460889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.460895 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:12:01.460913 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.460914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.460925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.464249 (+ 3324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.464286 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.465004 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.466032 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.466343 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.466677 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.467717 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.467718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.467719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.467725 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.469406 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.469840 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.469841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.469842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.470281 (+ 439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470521 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.470522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.470522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.470656 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.471131 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.471170 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.471253 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2562,"thread_start_us":55,"threads_started":1}
I20260925 12:12:01.471525 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.456656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42921 (local address 127.0.98.62:40957)
0925 12:12:01.456752 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.456755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.456947 (+ 192us) server_negotiation.cc:407] Connection header received
0925 12:12:01.457693 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.457695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.457707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.457741 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.458795 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459561 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.460723 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.461014 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.461056 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.465649 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.465650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.465650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.465657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.469185 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470631 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.470632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.470632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.470693 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470902 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.470903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.470903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.471041 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.471088 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.471327 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.471404 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 12:12:01.471853 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.456666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:42921)
0925 12:12:01.456833 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.456873 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.457572 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.457889 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.457893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.458618 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.458680 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.458685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459779 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.459780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.460622 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.460624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.461132 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.461133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.461199 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.462279 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.462285 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.465328 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.470156 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.470158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.470534 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.470770 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.470772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.470828 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.471139 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.471140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.471198 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.471631 (+ 433us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.471738 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260925 12:12:01.472218 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40957
I20260925 12:12:01.472268 7733 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.472339 7733 heartbeater.cc:511] Master 127.0.98.62:40957 requested a full tablet report, sending...
I20260925 12:12:01.472394 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.457120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50967 (local address 127.0.98.60:44431)
0925 12:12:01.457187 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.457189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.457366 (+ 177us) server_negotiation.cc:407] Connection header received
0925 12:12:01.457693 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.457695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.457707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.457749 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.458801 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459561 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.460722 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.461021 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.461057 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.464619 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.464620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.464620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.464627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.468275 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.471370 (+ 3095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.471371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.471371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.471422 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.471824 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.471825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.471825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.471960 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.472018 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.472219 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.472305 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 12:12:01.472611 6493 ts_manager.cc:194] Re-registered known tserver with Master: 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165)
I20260925 12:12:01.472765 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.456960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44431 (local address 127.0.98.3:50967)
0925 12:12:01.457269 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.457278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.457582 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.458262 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.458263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.458412 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:01.458680 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.458685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.459658 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.459659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.460623 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.460624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.461134 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.461135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.461199 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.462319 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:01.462324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:01.464513 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:01.470962 (+ 6449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.470963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.470963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.471178 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.471703 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:01.471704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:01.471705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:01.471765 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:01.472045 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:01.472045 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:01.472099 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.472562 (+ 463us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.472664 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":45,"thread_start_us":58,"threads_started":1}
I20260925 12:12:01.473094 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42921
I20260925 12:12:01.473383 7732 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44431
I20260925 12:12:01.473434 7732 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:01.473495 7732 heartbeater.cc:511] Master 127.0.98.60:44431 requested a full tablet report, sending...
I20260925 12:12:01.473744 7276 ts_manager.cc:194] Registered new tserver with Master: 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165)
I20260925 12:12:01.473897 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.458795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.98.60:44431)
0925 12:12:01.460840 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.460843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.461398 (+ 555us) server_negotiation.cc:407] Connection header received
0925 12:12:01.461451 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.461452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.461463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.461529 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.462507 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.463276 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.464356 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.464684 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.464724 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.466988 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.466989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.466990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.466996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.470883 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.471861 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.471862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.471862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.471905 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.472269 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.472270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.472271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.472385 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.473217 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.473447 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.473527 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2028,"thread_start_us":50,"threads_started":1}
I20260925 12:12:01.474977 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.458494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.98.62:40957)
0925 12:12:01.460789 (+ 2295us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.460791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.461222 (+ 431us) server_negotiation.cc:407] Connection header received
0925 12:12:01.461489 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.461491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.461504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.461549 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.465581 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.468042 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.469230 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.470022 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.470106 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.472111 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.472112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.472112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.472118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.474048 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.474422 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.474424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.474424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.474488 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.474647 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.474648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.474648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.474737 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.474800 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.474862 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.474920 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":73,"threads_started":1}
I20260925 12:12:01.476379 6493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:12:01.478084 6493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:01.485975 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.481895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43165 (local address 127.0.0.1:58838)
0925 12:12:01.481931 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.481943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.481981 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.482597 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.482598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.482599 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.482650 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.482651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.483922 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.483923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.485032 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.485034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485722 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.485789 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.485792 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.485832 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.485878 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":4}
I20260925 12:12:01.486140 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.481993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46681 (local address 127.0.0.1:60450)
0925 12:12:01.482178 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.482188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.482275 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.482669 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.482670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.482672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.482719 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.482720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.483596 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.483597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.484812 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.484814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485887 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.485954 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.485958 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.486003 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.486049 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":19,"thread_start_us":104,"threads_started":1}
I20260925 12:12:01.486140 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.481982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34509 (local address 127.0.0.1:49478)
0925 12:12:01.482100 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.482109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.482148 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.482857 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.482858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.482860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.482891 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.482891 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.484065 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.484065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.485086 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.485089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485945 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.486015 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.486018 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.486045 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.486071 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":16,"thread_start_us":78,"threads_started":1}
I20260925 12:12:01.486451 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.481973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.0.98.3:43165)
0925 12:12:01.482421 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.482426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.482435 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:01.482473 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.482476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.482493 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.482599 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.482729 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.483835 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485115 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485629 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.486322 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.486349 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.486398 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":104,"threads_started":1}
I20260925 12:12:01.486465 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.482237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.98.1:46681)
0925 12:12:01.482431 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.482435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.482446 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:12:01.482488 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.482490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.482505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.482671 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.482781 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.483503 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.484961 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485692 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485776 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.485851 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.486350 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.486395 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260925 12:12:01.487005 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.482186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.98.2:34509)
0925 12:12:01.482715 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.482719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.482727 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:01.482765 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.482767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.482784 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.482857 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.482944 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.483986 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.485159 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.485859 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.486788 (+ 929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.486868 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.486892 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.486933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":87,"threads_started":1}
I20260925 12:12:01.487108 7393 tablet_service.cc:1467] Processing CreateTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b (DEFAULT_TABLE table=test-table [id=332be854c72a449d97c1e6799c480b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:01.487131 7666 tablet_service.cc:1467] Processing CreateTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b (DEFAULT_TABLE table=test-table [id=332be854c72a449d97c1e6799c480b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:01.487432 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e1126a0eb04c64b6d36a86a3256e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:01.487449 7666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e1126a0eb04c64b6d36a86a3256e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:01.487560 7530 tablet_service.cc:1467] Processing CreateTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b (DEFAULT_TABLE table=test-table [id=332be854c72a449d97c1e6799c480b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:01.487823 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e1126a0eb04c64b6d36a86a3256e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:01.490010 7758 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Bootstrap starting.
I20260925 12:12:01.490566 7758 tablet_bootstrap.cc:654] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:01.490859 7758 log.cc:824] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:01.490875 7759 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Bootstrap starting.
I20260925 12:12:01.491475 7760 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Bootstrap starting.
I20260925 12:12:01.491921 7759 tablet_bootstrap.cc:654] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:01.492072 7758 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: No bootstrap required, opened a new log
I20260925 12:12:01.492108 7760 tablet_bootstrap.cc:654] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:01.492151 7758 ts_tablet_manager.cc:1397] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:01.492235 7759 log.cc:824] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:01.492413 7760 log.cc:824] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:01.493100 7759 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: No bootstrap required, opened a new log
I20260925 12:12:01.493168 7759 ts_tablet_manager.cc:1397] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:12:01.494755 7758 raft_consensus.cc:348] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.494931 7758 raft_consensus.cc:374] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:01.494971 7758 raft_consensus.cc:729] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc345b8b302144dab0ff926d26851f2f, State: Initialized, Role: FOLLOWER
I20260925 12:12:01.495087 7758 consensus_queue.cc:260] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.495142 7759 raft_consensus.cc:348] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.495317 7759 raft_consensus.cc:374] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:01.495352 7758 ts_tablet_manager.cc:1428] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 12:12:01.495363 7759 raft_consensus.cc:729] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df81531e1b3455bbfd703ee9b62ecd6, State: Initialized, Role: FOLLOWER
I20260925 12:12:01.495428 7460 heartbeater.cc:503] Master 127.0.98.62:40957 was elected leader, sending a full tablet report...
I20260925 12:12:01.495466 7759 consensus_queue.cc:260] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.495718 7759 ts_tablet_manager.cc:1428] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:12:01.496093 7733 heartbeater.cc:503] Master 127.0.98.62:40957 was elected leader, sending a full tablet report...
I20260925 12:12:01.497076 7760 tablet_bootstrap.cc:492] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: No bootstrap required, opened a new log
I20260925 12:12:01.497148 7760 ts_tablet_manager.cc:1397] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 12:12:01.499151 7760 raft_consensus.cc:348] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.499279 7760 raft_consensus.cc:374] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:01.499300 7760 raft_consensus.cc:729] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9884e2074344ea95cc53766d2ee9be, State: Initialized, Role: FOLLOWER
I20260925 12:12:01.499434 7760 consensus_queue.cc:260] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.499665 7760 ts_tablet_manager.cc:1428] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:12:01.499743 7597 heartbeater.cc:503] Master 127.0.98.62:40957 was elected leader, sending a full tablet report...
I20260925 12:12:01.521162 7766 raft_consensus.cc:482] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:01.521346 7766 raft_consensus.cc:504] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.521731 7766 leader_election.cc:288] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165), fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:12:01.524478 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.521994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35253 (local address 127.0.98.1:46681)
0925 12:12:01.522047 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.522049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.522058 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:01.522087 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.522088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.522100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.522155 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.522330 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.522802 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.523744 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.524107 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.524201 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.524212 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.524371 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.524407 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:01.524513 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.521901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46681 (local address 127.0.98.2:35253)
0925 12:12:01.521967 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.521981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.522033 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.522171 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.522172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.522173 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.522258 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.522260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.522888 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.522889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.523646 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.523648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.524202 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.524203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.524281 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.524290 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.524351 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.524391 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 12:12:01.524533 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.522018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58321 (local address 127.0.98.3:43165)
0925 12:12:01.522062 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.522064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.522072 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:01.522100 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.522100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.522113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.522185 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:01.522344 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.522801 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.523793 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.524145 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.524230 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.524239 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.524405 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.524443 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:12:01.524587 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.521901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43165 (local address 127.0.98.2:58321)
0925 12:12:01.521995 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:01.522006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:01.522045 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:01.522187 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:01.522187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:01.522189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:01.522269 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.522270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.522887 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.522888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.523679 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:01.523681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.524228 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.524229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:01.524320 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.524323 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:01.524398 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:12:01.524436 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":120}
I20260925 12:12:01.524902 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e1126a0eb04c64b6d36a86a3256e0b" candidate_uuid: "cf9884e2074344ea95cc53766d2ee9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc345b8b302144dab0ff926d26851f2f" is_pre_election: true
I20260925 12:12:01.525084 7413 raft_consensus.cc:2454] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9884e2074344ea95cc53766d2ee9be in term 0.
I20260925 12:12:01.525098 7686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e1126a0eb04c64b6d36a86a3256e0b" candidate_uuid: "cf9884e2074344ea95cc53766d2ee9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" is_pre_election: true
I20260925 12:12:01.525243 7686 raft_consensus.cc:2454] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9884e2074344ea95cc53766d2ee9be in term 0.
I20260925 12:12:01.525333 7486 leader_election.cc:302] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf9884e2074344ea95cc53766d2ee9be, fc345b8b302144dab0ff926d26851f2f; no voters:
I20260925 12:12:01.525476 7766 raft_consensus.cc:2781] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:01.525528 7766 raft_consensus.cc:482] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:01.525552 7766 raft_consensus.cc:3037] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:01.526297 7766 raft_consensus.cc:504] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.526444 7766 leader_election.cc:288] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [CANDIDATE]: Term 1 election: Requested vote from peers 7df81531e1b3455bbfd703ee9b62ecd6 (127.0.98.3:43165), fc345b8b302144dab0ff926d26851f2f (127.0.98.1:46681)
I20260925 12:12:01.526635 7686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e1126a0eb04c64b6d36a86a3256e0b" candidate_uuid: "cf9884e2074344ea95cc53766d2ee9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df81531e1b3455bbfd703ee9b62ecd6"
I20260925 12:12:01.526670 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e1126a0eb04c64b6d36a86a3256e0b" candidate_uuid: "cf9884e2074344ea95cc53766d2ee9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc345b8b302144dab0ff926d26851f2f"
I20260925 12:12:01.526727 7686 raft_consensus.cc:3037] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:01.526737 7413 raft_consensus.cc:3037] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:01.527596 7686 raft_consensus.cc:2454] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9884e2074344ea95cc53766d2ee9be in term 1.
I20260925 12:12:01.527596 7413 raft_consensus.cc:2454] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9884e2074344ea95cc53766d2ee9be in term 1.
I20260925 12:12:01.527782 7483 leader_election.cc:302] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df81531e1b3455bbfd703ee9b62ecd6, cf9884e2074344ea95cc53766d2ee9be; no voters:
I20260925 12:12:01.527897 7766 raft_consensus.cc:2781] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:01.528074 7766 raft_consensus.cc:686] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 1 LEADER]: Becoming Leader. State: Replica: cf9884e2074344ea95cc53766d2ee9be, State: Running, Role: LEADER
I20260925 12:12:01.528182 7766 consensus_queue.cc:237] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } }
I20260925 12:12:01.529049 6493 catalog_manager.cc:5495] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be reported cstate change: term changed from 0 to 1, leader changed from <none> to cf9884e2074344ea95cc53766d2ee9be (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cf9884e2074344ea95cc53766d2ee9be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf9884e2074344ea95cc53766d2ee9be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 } health_report { overall_health: UNKNOWN } } }
I20260925 12:12:01.545133 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.542617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.0.98.2:34509)
0925 12:12:01.542672 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.542674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.542770 (+ 96us) server_negotiation.cc:407] Connection header received
0925 12:12:01.542802 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.542803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.542821 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.542875 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:01.543054 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.543602 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.544293 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.544471 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.544573 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.544798 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:01.544913 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:01.545008 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.545027 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.545048 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:12:01.546921 7686 raft_consensus.cc:1261] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 1 FOLLOWER]: Refusing update from remote peer cf9884e2074344ea95cc53766d2ee9be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:01.546947 7413 raft_consensus.cc:1261] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 1 FOLLOWER]: Refusing update from remote peer cf9884e2074344ea95cc53766d2ee9be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:01.547237 7766 consensus_queue.cc:1035] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df81531e1b3455bbfd703ee9b62ecd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:01.547403 7767 consensus_queue.cc:1035] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc345b8b302144dab0ff926d26851f2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:01.552028 7771 mvcc.cc:205] Tried to move back new op lower bound from 7333225765053689856 to 7333225764980301824. Current Snapshot: MvccSnapshot[applied={T|T < 7333225765053689856 or (T in {7333225765053689856})}]
I20260925 12:12:01.552520 6493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:01.552613 6493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:01.552763 7772 mvcc.cc:205] Tried to move back new op lower bound from 7333225765053689856 to 7333225764980301824. Current Snapshot: MvccSnapshot[applied={T|T < 7333225765053689856}]
I20260925 12:12:01.553838 6493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba85b6019884f15a1119d6a183f1763: Sending DeleteTablet for 3 replicas of tablet f7e1126a0eb04c64b6d36a86a3256e0b
I20260925 12:12:01.554157 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60450
I20260925 12:12:01.554196 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49478
I20260925 12:12:01.554519 7780 tablet_replica.cc:330] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: stopping tablet replica
I20260925 12:12:01.554625 7780 raft_consensus.cc:2229] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:01.554683 7666 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260925 12:12:01.554730 7780 raft_consensus.cc:2258] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:01.554850 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e1126a0eb04c64b6d36a86a3256e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260925 12:12:01.555260 7782 tablet_replica.cc:330] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: stopping tablet replica
I20260925 12:12:01.555341 7780 ts_tablet_manager.cc:1905] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:01.555357 7782 raft_consensus.cc:2229] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:01.555449 7782 raft_consensus.cc:2258] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:01.555763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6461
I20260925 12:12:01.557322 7780 ts_tablet_manager.cc:1918] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:01.557415 7780 log.cc:1196] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/f7e1126a0eb04c64b6d36a86a3256e0b
I20260925 12:12:01.557680 7780 ts_tablet_manager.cc:1939] T f7e1126a0eb04c64b6d36a86a3256e0b P fc345b8b302144dab0ff926d26851f2f: Deleting consensus metadata
W20260925 12:12:01.562533 7347 connection.cc:535] server connection from 127.0.0.1:60450 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:12:01.563320 7781 tablet_replica.cc:330] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: stopping tablet replica
I20260925 12:12:01.563429 7781 raft_consensus.cc:2229] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:01.563531 7781 raft_consensus.cc:2258] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:01.564028 7781 ts_tablet_manager.cc:1905] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:01.564785 7782 ts_tablet_manager.cc:1905] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:01.567646 7781 ts_tablet_manager.cc:1918] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:01.567726 7781 log.cc:1196] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/f7e1126a0eb04c64b6d36a86a3256e0b
I20260925 12:12:01.568012 7781 ts_tablet_manager.cc:1939] T f7e1126a0eb04c64b6d36a86a3256e0b P cf9884e2074344ea95cc53766d2ee9be: Deleting consensus metadata
I20260925 12:12:01.568181 7782 ts_tablet_manager.cc:1918] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:01.568238 7782 log.cc:1196] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/f7e1126a0eb04c64b6d36a86a3256e0b
I20260925 12:12:01.568545 7782 ts_tablet_manager.cc:1939] T f7e1126a0eb04c64b6d36a86a3256e0b P 7df81531e1b3455bbfd703ee9b62ecd6: Deleting consensus metadata
W20260925 12:12:01.568923 7619 connection.cc:441] server connection from 127.0.0.1:58838 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58838 (request call id 1) could send its response
W20260925 12:12:01.569116 7619 connection.cc:441] server connection from 127.0.0.1:58838 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58838 (request call id 2) could send its response
I20260925 12:12:01.569144 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.554798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.0.98.60:44431)
0925 12:12:01.554941 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.554943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.560671 (+ 5728us) server_negotiation.cc:407] Connection header received
0925 12:12:01.560794 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.560795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.560818 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.560884 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.561618 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.562414 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.563687 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.564014 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.564054 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.565775 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.565776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.565776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.565783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.567492 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.567914 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.567915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.567916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.567968 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.568182 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.568183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.568183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.568318 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.568391 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.568627 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.569003 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260925 12:12:01.569430 7484 connection.cc:441] server connection from 127.0.0.1:49478 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49478 (request call id 1) could send its response
I20260925 12:12:01.573211 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.564558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.0.98.60:44431)
0925 12:12:01.564597 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.564599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.565287 (+ 688us) server_negotiation.cc:407] Connection header received
0925 12:12:01.565377 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.565377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.565387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.565416 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.566080 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.566755 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.567961 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.568249 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.568298 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.570378 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.570380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.570380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.570386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.572028 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.572416 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.572417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.572417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.572472 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.572706 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.572707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.572708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.572819 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.572921 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.573026 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.573093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:12:01.575600 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:01.564920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.98.61:35463)
0925 12:12:01.564983 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:01.564985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:01.568617 (+ 3632us) server_negotiation.cc:407] Connection header received
0925 12:12:01.568719 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:01.568720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:01.568736 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:01.568768 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:01.569720 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.570501 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.571483 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:01.571697 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:01.571775 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:01.573109 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:01.573110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:01.573111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:01.573118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:01.574581 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.574887 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.574888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.574889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.574946 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:01.575114 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:01.575115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:01.575115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:01.575251 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:01.575348 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:01.575419 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:12:01.575503 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:12:02.552239 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:02.551899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:36887)
0925 12:12:02.552103 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:02.552193 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
W20260925 12:12:02.552467 7460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40957 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:02.554680 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:02.554423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:33437)
0925 12:12:02.554532 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:02.554633 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
W20260925 12:12:02.554931 7733 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40957 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:02.555048 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:02.554747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:52669)
0925 12:12:02.554937 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:02.555005 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20260925 12:12:02.555220 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40957 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:03.054311 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:03.054457 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:03.054726 7794 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:03.055055 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:03.054870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50378)
0925 12:12:03.054970 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:03.055023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":60,"threads_started":1}
W20260925 12:12:03.055210 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:03.055631 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:03.055539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50392)
0925 12:12:03.055590 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:03.055615 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 12:12:03.055721 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:03.055763 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:03.055843 7794 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:03.553261 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:03.552943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:36005)
0925 12:12:03.553164 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:03.553216 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260925 12:12:03.555544 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:03.555287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:38483)
0925 12:12:03.555390 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:03.555490 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260925 12:12:03.555809 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:03.555547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:33547)
0925 12:12:03.555729 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:03.555775 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260925 12:12:04.554060 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:04.553735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:52453)
0925 12:12:04.553955 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:04.554001 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260925 12:12:04.556119 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:04.555920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:54565)
0925 12:12:04.556033 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:04.556074 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260925 12:12:04.556402 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:04.556184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:56839)
0925 12:12:04.556321 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:04.556346 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260925 12:12:05.054688 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:05.054795 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:05.054979 7803 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:05.055397 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:05.055193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50686)
0925 12:12:05.055317 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:05.055354 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260925 12:12:05.055975 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:05.055734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50696)
0925 12:12:05.055921 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:05.055949 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:12:05.056043 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:05.056087 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:05.056151 7803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:05.554733 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:05.554507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:43811)
0925 12:12:05.554637 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:05.554680 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260925 12:12:05.556805 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:05.556594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:44381)
0925 12:12:05.556688 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:05.556721 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":55,"threads_started":1}
I20260925 12:12:05.557126 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:05.556845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:53249)
0925 12:12:05.557031 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:05.557063 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":59,"threads_started":1}
I20260925 12:12:06.555487 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:06.555255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:52681)
0925 12:12:06.555391 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:06.555441 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260925 12:12:06.557507 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:06.557300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:40843)
0925 12:12:06.557430 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:06.557469 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260925 12:12:06.557780 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:06.557488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:52355)
0925 12:12:06.557657 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:06.557716 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":38,"threads_started":1}
I20260925 12:12:07.384477 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:07.384615 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:07.384781 7812 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:07.385139 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:07.384952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50886)
0925 12:12:07.385059 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:07.385102 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260925 12:12:07.385586 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:07.385496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50898)
0925 12:12:07.385542 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:07.385567 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:12:07.385708 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:07.385775 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:07.385865 7812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:07.556267 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:07.556015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:60665)
0925 12:12:07.556155 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:07.556206 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260925 12:12:07.558243 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:07.558035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:45463)
0925 12:12:07.558134 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:07.558175 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
I20260925 12:12:07.558445 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:07.558241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:40677)
0925 12:12:07.558371 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:07.558401 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260925 12:12:08.557309 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:08.556964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:54839)
0925 12:12:08.557166 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:08.557232 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":64,"threads_started":1}
I20260925 12:12:08.558816 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:08.558592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:33763)
0925 12:12:08.558739 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:08.558771 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260925 12:12:08.559010 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:08.558815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:49157)
0925 12:12:08.558939 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:08.558968 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260925 12:12:09.558130 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:09.557870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:43681)
0925 12:12:09.558004 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:09.558074 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260925 12:12:09.559505 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:09.559199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:54591)
0925 12:12:09.559410 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:09.559453 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260925 12:12:09.559681 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:09.559458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:33777)
0925 12:12:09.559567 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:09.559626 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260925 12:12:10.558825 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:10.558573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:36511)
0925 12:12:10.558717 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:10.558760 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260925 12:12:10.560256 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:10.560063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:34521)
0925 12:12:10.560188 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:10.560222 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260925 12:12:10.560322 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:10.560009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:58583)
0925 12:12:10.560126 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:10.560259 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260925 12:12:11.559516 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:11.559258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:48351)
0925 12:12:11.559404 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:11.559456 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260925 12:12:11.560789 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:11.560613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:45255)
0925 12:12:11.560713 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:11.560755 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260925 12:12:11.560871 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:11.560668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:46841)
0925 12:12:11.560774 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:11.560805 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260925 12:12:12.431366 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:12.431464 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:12.431694 7830 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:12.432080 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:12.431872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50190)
0925 12:12:12.431984 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:12.432028 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
W20260925 12:12:12.432170 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260925 12:12:12.432590 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:12.432437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50194)
0925 12:12:12.432527 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:12.432560 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:12:12.432706 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:12.432758 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:12.432821 7830 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:12.560382 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:12.560026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:35003)
0925 12:12:12.560284 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:12.560338 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":135,"threads_started":1}
I20260925 12:12:12.561383 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:12.561183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:57423)
0925 12:12:12.561296 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:12.561335 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260925 12:12:12.561445 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:12.561243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:57185)
0925 12:12:12.561356 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:12.561392 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260925 12:12:13.561120 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:13.560789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:56997)
0925 12:12:13.560984 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:13.561056 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260925 12:12:13.561928 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:13.561737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:54749)
0925 12:12:13.561865 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:13.561894 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260925 12:12:13.562099 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:13.561917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:39911)
0925 12:12:13.562023 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:13.562056 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260925 12:12:14.561837 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:14.561620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:36945)
0925 12:12:14.561737 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:14.561778 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260925 12:12:14.562623 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:14.562456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:50959)
0925 12:12:14.562561 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:14.562589 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260925 12:12:14.562625 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:14.562403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:45295)
0925 12:12:14.562550 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:14.562588 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260925 12:12:15.562741 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:15.562377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:46311)
0925 12:12:15.562540 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:15.562631 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260925 12:12:15.563194 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:15.563037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:36335)
0925 12:12:15.563130 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:15.563153 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20260925 12:12:15.563211 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:15.563026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:39075)
0925 12:12:15.563127 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:15.563162 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":65,"threads_started":1}
I20260925 12:12:16.563560 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:16.563244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:40295)
0925 12:12:16.563421 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:16.563471 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260925 12:12:16.563835 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:16.563591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:56313)
0925 12:12:16.563734 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:16.563792 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260925 12:12:16.563925 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:16.563622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:49931)
0925 12:12:16.563763 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:16.563808 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":49,"threads_started":1}
I20260925 12:12:17.564399 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:17.564148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:39777)
0925 12:12:17.564302 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:17.564343 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260925 12:12:17.564402 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:17.564216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:52053)
0925 12:12:17.564324 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:17.564358 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":47,"threads_started":1}
I20260925 12:12:17.564402 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:17.564194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:48689)
0925 12:12:17.564324 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:17.564359 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 12:12:17.972891 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:17.972978 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:17.973239 7851 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:17.973582 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:17.973398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50482)
0925 12:12:17.973501 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:17.973540 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
W20260925 12:12:17.973668 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:17.974006 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:17.973908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50486)
0925 12:12:17.973958 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:17.973983 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:12:17.974087 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:17.974133 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:17.974195 7851 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:18.565168 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:18.564943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:47791)
0925 12:12:18.565068 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:18.565120 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260925 12:12:18.565168 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:18.564959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:33979)
0925 12:12:18.565076 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:18.565117 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260925 12:12:18.565182 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:18.564918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:56585)
0925 12:12:18.565071 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:18.565134 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260925 12:12:19.565801 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:19.565586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:38809)
0925 12:12:19.565732 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:19.565763 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260925 12:12:19.565801 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:19.565588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:56389)
0925 12:12:19.565708 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:19.565748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 12:12:19.565832 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:19.565643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:58251)
0925 12:12:19.565732 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:19.565793 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":46,"threads_started":1}
I20260925 12:12:20.566452 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:20.566217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:33933)
0925 12:12:20.566351 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:20.566394 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 12:12:20.566455 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:20.566215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:49255)
0925 12:12:20.566376 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:20.566411 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260925 12:12:20.566455 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:20.566218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:50827)
0925 12:12:20.566387 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:20.566411 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260925 12:12:20.962333 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:20.962419 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:20.962582 7863 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:20.962911 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:20.962710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50630)
0925 12:12:20.962835 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:20.962874 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 12:12:20.963351 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:20.963255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:50638)
0925 12:12:20.963307 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:20.963333 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 12:12:20.963457 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:20.963514 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:20.963579 7863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:21.567047 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:21.566817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:59671)
0925 12:12:21.566950 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:21.566990 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260925 12:12:21.567106 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:21.566853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:35427)
0925 12:12:21.567036 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:21.567066 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":39,"threads_started":1}
I20260925 12:12:21.567178 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:21.566816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:34553)
0925 12:12:21.566950 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:21.566990 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260925 12:12:22.567761 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:22.567564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:50799)
0925 12:12:22.567688 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:22.567718 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
I20260925 12:12:22.567761 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:22.567545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:37531)
0925 12:12:22.567663 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:22.567710 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260925 12:12:22.568104 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:22.567897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:56139)
0925 12:12:22.568027 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:22.568068 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 12:12:23.568467 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:23.568200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:43321)
0925 12:12:23.568365 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:23.568406 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260925 12:12:23.568522 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:23.568298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:51943)
0925 12:12:23.568405 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:23.568448 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260925 12:12:23.568754 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:23.568517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:34475)
0925 12:12:23.568680 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:23.568710 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260925 12:12:24.569202 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:24.568966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:43919)
0925 12:12:24.569122 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:24.569161 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260925 12:12:24.569206 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:24.568944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:33649)
0925 12:12:24.569111 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:24.569153 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260925 12:12:24.569274 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:24.569114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:58971)
0925 12:12:24.569206 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:24.569232 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
I20260925 12:12:25.569860 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:25.569603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:42053)
0925 12:12:25.569733 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:25.569772 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260925 12:12:25.570034 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:25.569610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:49801)
0925 12:12:25.569945 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:25.569983 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":55,"threads_started":1}
I20260925 12:12:25.570112 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:25.569610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:52127)
0925 12:12:25.570041 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:25.570072 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":55,"threads_started":1}
I20260925 12:12:26.570883 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:26.570475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:43315)
0925 12:12:26.570774 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:26.570824 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":60,"threads_started":1}
I20260925 12:12:26.570915 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:26.570729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:53323)
0925 12:12:26.570830 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:26.570864 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":42,"threads_started":1}
I20260925 12:12:26.571189 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:26.570678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:60891)
0925 12:12:26.571051 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:26.571087 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":76,"threads_started":1}
I20260925 12:12:27.571787 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:27.571522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:35045)
0925 12:12:27.571692 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:27.571732 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260925 12:12:27.571800 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:27.571567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:37839)
0925 12:12:27.571741 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:27.571767 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260925 12:12:27.571851 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:27.571599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:45551)
0925 12:12:27.571760 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:27.571795 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260925 12:12:28.572477 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:28.572239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:57537)
0925 12:12:28.572371 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:28.572415 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 12:12:28.572615 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:28.572393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:45845)
0925 12:12:28.572536 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:28.572572 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":46,"threads_started":1}
I20260925 12:12:28.572738 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:28.572361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:56387)
0925 12:12:28.572652 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:28.572687 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":74,"threads_started":1}
I20260925 12:12:28.976533 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bba85b6019884f15a1119d6a183f1763)
I20260925 12:12:28.976627 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bba85b6019884f15a1119d6a183f1763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40957 } } peers { permanent_uuid: "fcba8cf5dc0c414ca1124072fcb6dbf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35463 } attrs { promote: false } } peers { permanent_uuid: "0b413adb1ec4460d94fa3811ef4d924e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44431 } attrs { promote: true } }
I20260925 12:12:28.976751 7890 leader_election.cc:288] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957)
I20260925 12:12:28.977133 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:28.976932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:45586)
0925 12:12:28.977051 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:28.977095 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
W20260925 12:12:28.977221 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 12:12:28.977594 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:28.977500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.0.1:45592)
0925 12:12:28.977548 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:28.977576 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:12:28.977681 7080 leader_election.cc:334] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bba85b6019884f15a1119d6a183f1763 (127.0.98.62:40957): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
I20260925 12:12:28.977735 7080 leader_election.cc:302] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: fcba8cf5dc0c414ca1124072fcb6dbf7; no voters: bba85b6019884f15a1119d6a183f1763
I20260925 12:12:28.977821 7890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fcba8cf5dc0c414ca1124072fcb6dbf7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:12:29.573241 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:29.573025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:58049)
0925 12:12:29.573144 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:29.573182 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260925 12:12:29.573297 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:29.573035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:54485)
0925 12:12:29.573198 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:29.573236 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260925 12:12:29.573635 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:29.573405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:55221)
0925 12:12:29.573556 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:29.573588 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":42,"threads_started":1}
I20260925 12:12:30.573915 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:30.573665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:59863)
0925 12:12:30.573794 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:30.573855 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260925 12:12:30.574023 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:30.573842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:53685)
0925 12:12:30.573941 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:30.573975 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260925 12:12:30.574229 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:30.573953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:47405)
0925 12:12:30.574083 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:30.574117 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260925 12:12:31.574707 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:31.574469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.3:47353)
0925 12:12:31.574600 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:31.574644 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 12:12:31.574765 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:31.574523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.2:37405)
0925 12:12:31.574678 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:31.574718 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260925 12:12:31.575026 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:31.574673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40957 (local address 127.0.98.1:59591)
0925 12:12:31.574925 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:31.574959 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":43,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:40957,127.0.98.61:35463,127.0.98.60:44431, num_attempts: 1)
I20260925 12:12:31.759725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 7240
I20260925 12:12:31.764631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 7467
I20260925 12:12:31.770493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 7603
I20260925 12:12:31.775499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 6958
I20260925 12:12:31.779944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 7147
2026-09-25T12:12:31Z chronyd exiting
I20260925 12:12:31.795770 392 test_util.cc:175] -----------------------------------------------
I20260925 12:12:31.795820 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790338290997044-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (34089 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:12:31 dist-test-slave-2hqq krb5kdc[7904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:12:31 dist-test-slave-2hqq krb5kdc[7904](info): set up 2 sockets
Sep 25 12:12:31 dist-test-slave-2hqq krb5kdc[7904](info): commencing operation
krb5kdc: starting...
W20260925 12:12:32.812865 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:12:32 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 12:12:32.931972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:33.013257 7917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:33.013453 7917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:33.013473 7917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:33.014549 7917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:33.014580 7917 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 12:12:33.014593 7917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:33.014606 7917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:33.014618 7917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:33.014631 7917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:33.016031 7917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:33.016216 7917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:33.016376 7917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:33.021188 7922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.021243 7925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.021205 7923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:33.021507 7917 server_base.cc:1034] running on GCE node
I20260925 12:12:33.021749 7917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:33.022037 7917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:33.023224 7917 hybrid_clock.cc:648] HybridClock initialized: now 1790338353023222 us; error 36 us; skew 500 ppm
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:33.025707 7917 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 12:12:33.026250 7917 webserver.cc:460] Webserver started at http://127.0.98.62:35633/ using document root <none> and password file <none>
I20260925 12:12:33.026489 7917 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:33.026559 7917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:33.026696 7917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:33.027733 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "e6c27d33cdbb4d6bad60b097d4a97e95"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "f75336f5b42b3f6be75c38ab893ec102"
server_key_iv: "4027b84b4976379d086e976fd698a9c5"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.028112 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6c27d33cdbb4d6bad60b097d4a97e95"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "f75336f5b42b3f6be75c38ab893ec102"
server_key_iv: "4027b84b4976379d086e976fd698a9c5"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.029492 7917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:12:33.030189 7932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.030455 7917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:33.030555 7917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e6c27d33cdbb4d6bad60b097d4a97e95"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "f75336f5b42b3f6be75c38ab893ec102"
server_key_iv: "4027b84b4976379d086e976fd698a9c5"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.030628 7917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:33.058126 7917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:33.060004 7917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:33.060190 7917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:33.064870 7917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35593
I20260925 12:12:33.064930 7994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35593 every 8 connection(s)
I20260925 12:12:33.065265 7917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:33.065860 7995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.066655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 7917
I20260925 12:12:33.066771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:12:33.066962 392 external_mini_cluster.cc:1428] Setting key dd791cdf9e011541cd761281a314eb28
I20260925 12:12:33.068864 7995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Bootstrap starting.
I20260925 12:12:33.069448 7995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.069741 7995 log.cc:824] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:33.070394 7995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: No bootstrap required, opened a new log
I20260925 12:12:33.071673 7995 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260925 12:12:33.071826 7995 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.071888 7995 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c27d33cdbb4d6bad60b097d4a97e95, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.072026 7995 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260925 12:12:33.072121 7995 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:33.072153 7995 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:33.072199 7995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.072818 7995 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260925 12:12:33.072902 7995 leader_election.cc:302] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c27d33cdbb4d6bad60b097d4a97e95; no voters:
I20260925 12:12:33.073081 7995 leader_election.cc:288] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:12:33.073127 8000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:33.073298 8000 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [term 1 LEADER]: Becoming Leader. State: Replica: e6c27d33cdbb4d6bad60b097d4a97e95, State: Running, Role: LEADER
I20260925 12:12:33.073323 7995 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:33.073462 8000 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260925 12:12:33.073858 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c27d33cdbb4d6bad60b097d4a97e95. Latest consensus state: current_term: 1 leader_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260925 12:12:33.074095 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:33.074033 8001 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27d33cdbb4d6bad60b097d4a97e95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.074496 8001 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:33.074676 8009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:33.075309 8009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:33.077165 8009 catalog_manager.cc:1347] Generated new cluster ID: dcf1e389ad4145e6bfa60775483b3b08
I20260925 12:12:33.077221 8009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:12:33.078593 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.067483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.98.62:35593)
0925 12:12:33.067668 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.067679 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.067718 (+ 39us) server_negotiation.cc:407] Connection header received
0925 12:12:33.068152 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.068158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.068211 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.068361 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.069073 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.070067 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.070898 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.071698 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.071834 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.075097 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.075100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.075100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.075110 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.077028 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.077479 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.077483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.077483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.077580 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.077797 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.077798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.077798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.078029 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.078205 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.078283 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.078418 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260925 12:12:33.091557 8009 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:12:33.092258 8009 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:33.098168 8009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Generated new TSK 0
I20260925 12:12:33.098338 8009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 12:12:33.127661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:33.226703 8021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:33.226936 8021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:33.226994 8021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:33.228135 8021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:33.228205 8021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:33.228241 8021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:12:33.228279 8021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:33.229785 8021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:33.230047 8021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:33.230270 8021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:33.235215 8027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.235210 8029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.235219 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:33.235570 8021 server_base.cc:1034] running on GCE node
I20260925 12:12:33.235778 8021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:33.236013 8021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:33.237174 8021 hybrid_clock.cc:648] HybridClock initialized: now 1790338353237173 us; error 39 us; skew 500 ppm
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:33.239478 8021 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:12:33.239876 8021 webserver.cc:460] Webserver started at http://127.0.98.1:44483/ using document root <none> and password file <none>
I20260925 12:12:33.240095 8021 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:33.240164 8021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:33.240314 8021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:33.241405 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "0c4147d7d8c676d1a9f9007d96b7d70e"
server_key_iv: "60eb8a5bbbddf15691f756423f953d11"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.241779 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "0c4147d7d8c676d1a9f9007d96b7d70e"
server_key_iv: "60eb8a5bbbddf15691f756423f953d11"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.243083 8021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:33.243762 8036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.243929 8021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:33.244027 8021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "0c4147d7d8c676d1a9f9007d96b7d70e"
server_key_iv: "60eb8a5bbbddf15691f756423f953d11"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.244125 8021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:33.259690 8021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:33.261576 8021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:33.261735 8021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:33.262243 8021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:33.262305 8021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.262348 8021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:33.262373 8021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.269119 8021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42411
I20260925 12:12:33.269183 8151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42411 every 8 connection(s)
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.269549 8021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:33.272168 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.262690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58713 (local address 127.0.98.62:35593)
0925 12:12:33.262757 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.262760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.262976 (+ 216us) server_negotiation.cc:407] Connection header received
0925 12:12:33.263453 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.263454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.263468 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.263562 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.264384 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.265264 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.266847 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.267212 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.267290 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.269922 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.269924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.269924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.269932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.271111 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.271461 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.271462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.271463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.271513 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.271685 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.271685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.271685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.271780 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.271828 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.272048 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.272108 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:12:33.272337 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.262678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:58713)
0925 12:12:33.262838 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.262875 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.263362 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.263610 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.263613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.263991 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.264276 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.264283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.265368 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.265369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.266747 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.266751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.267304 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.267305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.267401 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.267942 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.267952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.269813 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.271185 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.271190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.271192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.271395 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.271554 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.271554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.271555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.271635 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.271837 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.271842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.271946 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.272025 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.272146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260925 12:12:33.272821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8021
I20260925 12:12:33.272866 8152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260925 12:12:33.272935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:12:33.272944 8152 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:33.273119 8152 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260925 12:12:33.273154 392 external_mini_cluster.cc:1428] Setting key 266b6dfdf2ec5cfb83d32a57bc9dfd24
I20260925 12:12:33.273751 7948 ts_manager.cc:194] Registered new tserver with Master: a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411)
I20260925 12:12:33.274734 7948 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58713
I20260925 12:12:33.280126 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.273826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41371 (local address 127.0.98.62:35593)
0925 12:12:33.273860 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.273862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.274028 (+ 166us) server_negotiation.cc:407] Connection header received
0925 12:12:33.274100 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.274101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.274110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.274165 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.274599 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.275343 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.276431 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.276743 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.276809 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.277713 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.277715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.277715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.277722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.279047 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.279446 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.279446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.279447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.279492 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.279701 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.279701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.279702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.279789 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.279836 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.280005 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.280071 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:12:33.280252 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.273828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:41371)
0925 12:12:33.273969 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.273980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.274039 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.274202 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.274203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.274431 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.274488 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.274490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.275454 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.275455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.276344 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.276348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.276824 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.276825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.276873 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.277289 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.277294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.277619 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.279130 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.279131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.279132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.279371 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.279559 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.279560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.279561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.279633 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.279850 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.279851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.279905 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.279983 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.280091 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 12:12:33.297510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:33.377756 8161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:33.377957 8161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:33.377990 8161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:33.379136 8161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:33.379179 8161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:33.379199 8161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:12:33.379216 8161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:33.380668 8161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:33.380908 8161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:33.381145 8161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:33.386273 8169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.386353 8167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.386289 8166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:33.386533 8161 server_base.cc:1034] running on GCE node
I20260925 12:12:33.386724 8161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:33.386907 8161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:33.388038 8161 hybrid_clock.cc:648] HybridClock initialized: now 1790338353388034 us; error 33 us; skew 500 ppm
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:33.390415 8161 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:12:33.390856 8161 webserver.cc:460] Webserver started at http://127.0.98.2:43201/ using document root <none> and password file <none>
I20260925 12:12:33.391110 8161 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:33.391155 8161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:33.391317 8161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:33.392313 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "086e23ca752a4a11975c79ed7076286e"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "9174a0d7c29ed89704cfba25bd637cd0"
server_key_iv: "1f39f3893e673ee5ed03adaf194e4323"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.392684 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "086e23ca752a4a11975c79ed7076286e"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "9174a0d7c29ed89704cfba25bd637cd0"
server_key_iv: "1f39f3893e673ee5ed03adaf194e4323"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.393949 8161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:33.394651 8176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.394843 8161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.394924 8161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "086e23ca752a4a11975c79ed7076286e"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "9174a0d7c29ed89704cfba25bd637cd0"
server_key_iv: "1f39f3893e673ee5ed03adaf194e4323"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.395033 8161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:33.405122 8161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:33.406836 8161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:33.406983 8161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:33.407526 8161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:33.407575 8161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.407603 8161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:33.407627 8161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.414702 8161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37831
I20260925 12:12:33.414731 8291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37831 every 8 connection(s)
I20260925 12:12:33.415127 8161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:12:33.416846 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.407911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49993 (local address 127.0.98.62:35593)
0925 12:12:33.407972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.407974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.408187 (+ 213us) server_negotiation.cc:407] Connection header received
0925 12:12:33.408782 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.408784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.408797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.408848 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.409527 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.410055 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.411064 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.411274 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.411360 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.413992 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.413994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.413995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.414002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.415690 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.416030 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.416031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.416031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.416079 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.416294 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.416294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.416294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.416391 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.416440 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.416728 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.416787 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:12:33.417028 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.407912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:49993)
0925 12:12:33.408075 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.408107 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.408676 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.408917 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.408920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.409228 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.409418 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.409442 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.410156 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.410162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.410959 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.410961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.411368 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.411369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.411534 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.412034 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.412042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.413915 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.415756 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.415760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.415762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.415963 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.416122 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.416123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.416123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.416242 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.416445 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.416449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.416613 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.416698 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.416809 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260925 12:12:33.417454 8292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260925 12:12:33.417533 8292 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:33.417711 8292 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260925 12:12:33.418052 7949 ts_manager.cc:194] Registered new tserver with Master: 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831)
I20260925 12:12:33.418418 7949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49993
I20260925 12:12:33.421768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8161
I20260925 12:12:33.421887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:33.422116 392 external_mini_cluster.cc:1428] Setting key bb5e8afde8b4f2bd2ee5900f974956fa
I20260925 12:12:33.424099 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.418476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33959 (local address 127.0.98.62:35593)
0925 12:12:33.418549 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.418552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.418657 (+ 105us) server_negotiation.cc:407] Connection header received
0925 12:12:33.418726 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.418727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.418737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.418777 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.419105 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.419562 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.420265 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.420464 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.420605 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.421360 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.421362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.421362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.421369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.422928 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.423287 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.423288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.423289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.423340 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.423558 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.423559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.423559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.423706 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.423784 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.423935 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.424010 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:12:33.424113 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.418418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:33959)
0925 12:12:33.418607 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.418618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.418671 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.418791 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.418792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.418964 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.419021 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.419022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.419653 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.419654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.420173 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.420174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.420607 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.420608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.420673 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.421015 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.421019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.421274 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.423044 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.423045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.423046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.423216 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.423416 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.423417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.423418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.423485 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.423777 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.423778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.423821 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.423911 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.424008 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:12:33.449486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:33.531284 8301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:33.531520 8301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:33.531545 8301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:33.532814 8301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:33.532859 8301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:33.532881 8301 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 12:12:33.532899 8301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:33.534548 8301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:33.534792 8301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:33.535032 8301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:33.539885 8307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.539885 8309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:33.539978 8306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:33.540159 8301 server_base.cc:1034] running on GCE node
I20260925 12:12:33.540350 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:33.540602 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:33.541855 8301 hybrid_clock.cc:648] HybridClock initialized: now 1790338353541845 us; error 37 us; skew 500 ppm
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:33.544332 8301 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:12:33.544811 8301 webserver.cc:460] Webserver started at http://127.0.98.3:38255/ using document root <none> and password file <none>
I20260925 12:12:33.545044 8301 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:33.545116 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:33.545248 8301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:33.546274 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "0896f644a79546128dcf15119d04e1c5"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "b55b91c871e34c09a7642b7e31462c2c"
server_key_iv: "cc40a25f6cf1f4973a591a9e2e9e4bc3"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.546643 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0896f644a79546128dcf15119d04e1c5"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "b55b91c871e34c09a7642b7e31462c2c"
server_key_iv: "cc40a25f6cf1f4973a591a9e2e9e4bc3"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.548023 8301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:33.548787 8316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.548955 8301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:33.549053 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "0896f644a79546128dcf15119d04e1c5"
format_stamp: "Formatted at 2026-09-25 12:12:33 on dist-test-slave-2hqq"
server_key: "b55b91c871e34c09a7642b7e31462c2c"
server_key_iv: "cc40a25f6cf1f4973a591a9e2e9e4bc3"
server_key_version: "encryptionkey@0"
I20260925 12:12:33.549134 8301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:33.560680 8301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:33.562537 8301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:33.562700 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:33.563199 8301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:33.563264 8301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.563308 8301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:33.563336 8301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.569706 8301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42739
I20260925 12:12:33.569878 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42739 every 8 connection(s)
I20260925 12:12:33.570101 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.573333 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.563719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44439 (local address 127.0.98.62:35593)
0925 12:12:33.563761 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.563764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.563970 (+ 206us) server_negotiation.cc:407] Connection header received
0925 12:12:33.564533 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.564535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.564547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.564607 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.565426 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.566277 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.567868 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.568207 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.568290 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.570830 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.570832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.570832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.570839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.572004 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.572347 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.572348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.572348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.572393 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.572683 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.572684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.572684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.572821 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.572918 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.573201 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.573271 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:12:33.573427 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.563703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:44439)
0925 12:12:33.563851 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.563885 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.564393 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.564676 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.564681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.565021 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.565298 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.565305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.566389 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.566391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.567759 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.567762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.568297 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.568298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.568462 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.569019 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.569030 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.570746 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.572074 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.572079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.572081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.572283 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.572449 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.572450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.572450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.572627 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.572887 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.572892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.573053 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.573137 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.573253 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 12:12:33.573778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8301
I20260925 12:12:33.573913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:33.574060 8432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260925 12:12:33.574142 8432 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:33.574153 392 external_mini_cluster.cc:1428] Setting key 9f71bbe25bc966238d4e01541b6c0606
I20260925 12:12:33.574319 8432 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260925 12:12:33.574815 7949 ts_manager.cc:194] Registered new tserver with Master: 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:33.575330 7949 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44439
I20260925 12:12:33.580174 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.575251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49879 (local address 127.0.98.62:35593)
0925 12:12:33.575478 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.575480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.575487 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:33.575504 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.575505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.575513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.575560 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.575894 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.576348 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.577055 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.577225 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.577276 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.578019 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.578020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.578020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.578025 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.579361 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.579657 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.579658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.579658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.579702 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.579842 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.579842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.579842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.579921 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.579965 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.580069 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.580119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260925 12:12:33.580188 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.575167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:49879)
0925 12:12:33.575314 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.575324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.575373 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.575565 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.575566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.575754 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.575817 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.575819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.576434 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.576435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.576971 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.576972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.577291 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.577292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.577336 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.577679 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.577683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.577936 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.579435 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.579435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.579436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.579595 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.579758 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.579758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.579759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.579806 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.579974 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.579974 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.580005 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.580055 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.580123 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260925 12:12:33.586162 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:12:33.606421 7998 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:33.601532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.0.98.62:35593)
0925 12:12:33.601594 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.601596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.601604 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:33.601641 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.601641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.601649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.601701 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.602174 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.602634 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.603366 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.603535 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.603581 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.606323 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41306: BlockingRecv error: recv got EOF from 127.0.0.1:41306 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.613086 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.607158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.98.62:35593)
0925 12:12:33.607212 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.607215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.607299 (+ 84us) server_negotiation.cc:407] Connection header received
0925 12:12:33.607336 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.607336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.607344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.607381 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.607783 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.608218 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.608875 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.609049 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.609098 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.610981 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.610982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.610983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.610988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.612124 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.612403 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.612403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.612404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.612461 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.612682 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.612683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.612683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.612774 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.612855 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.612954 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.613004 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:12:33.614634 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:12:33.616197 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:33.622704 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37831 (local address 127.0.0.1:42212)
0925 12:12:33.618650 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.618694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.618764 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.619243 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.619246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.619248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.619312 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.619313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620249 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.620250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.621485 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.621487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622385 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.622470 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622484 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.622567 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.622619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":44,"threads_started":1}
I20260925 12:12:33.622910 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42739 (local address 127.0.0.1:59686)
0925 12:12:33.618947 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.618957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.618996 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.619388 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.619389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.619390 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.619427 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.619428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620706 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.620707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.621740 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.621742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622688 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.622751 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622755 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.622808 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.622850 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":27,"thread_start_us":69,"threads_started":1}
I20260925 12:12:33.622946 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.0.98.2:37831)
0925 12:12:33.619070 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.619074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.619084 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:33.619126 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.619128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.619140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.619239 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.619386 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620160 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.621572 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622294 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622372 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622445 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.622590 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.622630 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":80,"threads_started":1}
I20260925 12:12:33.623050 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.0.1:58486)
0925 12:12:33.618889 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.618900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.618957 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.619514 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.619515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.619517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.619562 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.619564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620764 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.620765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.621769 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.621771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622823 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.622883 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622887 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.622932 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.622975 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260925 12:12:33.623349 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.0.98.1:42411)
0925 12:12:33.618853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.618858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.619308 (+ 450us) server_negotiation.cc:407] Connection header received
0925 12:12:33.619352 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.619354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.619370 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.619449 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.619645 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620700 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.621861 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622654 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622719 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622784 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.623143 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.623194 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260925 12:12:33.623703 8226 tablet_service.cc:1467] Processing CreateTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c (DEFAULT_TABLE table=test-table [id=8dac6484694a4287b643311df016e0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.623955 8086 tablet_service.cc:1467] Processing CreateTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c (DEFAULT_TABLE table=test-table [id=8dac6484694a4287b643311df016e0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.624078 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a23af98b2cf49a1936ea1c33644cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.624269 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a23af98b2cf49a1936ea1c33644cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.624679 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.618931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.98.3:42739)
0925 12:12:33.619085 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.619090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.619099 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:33.619136 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.619139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.619155 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.619234 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.619482 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.620622 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.621822 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.622528 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.622588 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.622657 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.624569 (+ 1912us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.624610 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 12:12:33.625195 8366 tablet_service.cc:1467] Processing CreateTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c (DEFAULT_TABLE table=test-table [id=8dac6484694a4287b643311df016e0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.625509 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a23af98b2cf49a1936ea1c33644cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.628068 8460 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Bootstrap starting.
I20260925 12:12:33.628278 8461 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Bootstrap starting.
I20260925 12:12:33.628388 8462 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Bootstrap starting.
I20260925 12:12:33.628939 8461 tablet_bootstrap.cc:654] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.628966 8462 tablet_bootstrap.cc:654] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.629251 8462 log.cc:824] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:33.629251 8461 log.cc:824] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:33.630007 8461 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: No bootstrap required, opened a new log
I20260925 12:12:33.630024 8462 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: No bootstrap required, opened a new log
I20260925 12:12:33.630070 8460 tablet_bootstrap.cc:654] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.630085 8462 ts_tablet_manager.cc:1397] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:33.630084 8461 ts_tablet_manager.cc:1397] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:33.630380 8460 log.cc:824] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:33.631135 8460 tablet_bootstrap.cc:492] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: No bootstrap required, opened a new log
I20260925 12:12:33.631204 8460 ts_tablet_manager.cc:1397] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 12:12:33.631727 8462 raft_consensus.cc:348] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.631906 8462 raft_consensus.cc:374] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.631961 8462 raft_consensus.cc:729] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.632081 8462 consensus_queue.cc:260] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.632100 8461 raft_consensus.cc:348] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.632249 8461 raft_consensus.cc:374] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.632292 8461 raft_consensus.cc:729] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.632346 8462 ts_tablet_manager.cc:1428] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:12:33.632411 8461 consensus_queue.cc:260] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.632472 8432 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260925 12:12:33.632691 8461 ts_tablet_manager.cc:1428] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 12:12:33.632715 8292 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260925 12:12:33.633277 8460 raft_consensus.cc:348] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.633471 8460 raft_consensus.cc:374] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.633505 8460 raft_consensus.cc:729] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.633615 8460 consensus_queue.cc:260] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.633854 8460 ts_tablet_manager.cc:1428] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:12:33.633906 8152 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
W20260925 12:12:33.665656 8293 tablet.cc:2369] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:12:33.722958 8466 raft_consensus.cc:482] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:33.723136 8466 raft_consensus.cc:504] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.723583 8466 leader_election.cc:288] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411)
I20260925 12:12:33.726460 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.723851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40381 (local address 127.0.98.2:37831)
0925 12:12:33.723899 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.723902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.723913 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:12:33.723953 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.723954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.723966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.724038 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.724205 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.724769 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.725734 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.726105 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.726183 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.726194 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.726350 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.726388 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:12:33.726481 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.723786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37831 (local address 127.0.98.3:40381)
0925 12:12:33.723843 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.723855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.723902 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.724059 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.724060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.724062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.724132 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.724134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.724851 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.724852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.725636 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.725638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.726194 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.726195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.726278 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.726281 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.726345 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.726400 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":12}
I20260925 12:12:33.726576 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.723894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42577 (local address 127.0.98.1:42411)
0925 12:12:33.723980 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.723982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.724096 (+ 114us) server_negotiation.cc:407] Connection header received
0925 12:12:33.724157 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.724158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.724169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.724234 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.724398 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.724874 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.725824 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.726194 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.726275 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.726285 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.726457 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.726492 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 12:12:33.726763 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.723811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.98.3:42577)
0925 12:12:33.724017 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.724026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.724097 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.724253 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.724254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.724256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.724329 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.724331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.725003 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.725004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.725730 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.725731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.726291 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.726292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.726367 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.726370 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.726445 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.726482 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
I20260925 12:12:33.726784 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a23af98b2cf49a1936ea1c33644cf2c" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e" is_pre_election: true
I20260925 12:12:33.726975 8246 raft_consensus.cc:2454] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 0.
I20260925 12:12:33.727073 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a23af98b2cf49a1936ea1c33644cf2c" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" is_pre_election: true
I20260925 12:12:33.727229 8106 raft_consensus.cc:2454] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 0.
I20260925 12:12:33.727212 8320 leader_election.cc:302] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, 0896f644a79546128dcf15119d04e1c5; no voters:
I20260925 12:12:33.727366 8466 raft_consensus.cc:2781] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:33.727420 8466 raft_consensus.cc:482] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:33.727442 8466 raft_consensus.cc:3037] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.728327 8466 raft_consensus.cc:504] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.728461 8466 leader_election.cc:288] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 election: Requested vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411)
I20260925 12:12:33.728647 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a23af98b2cf49a1936ea1c33644cf2c" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
I20260925 12:12:33.728683 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a23af98b2cf49a1936ea1c33644cf2c" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e"
I20260925 12:12:33.728726 8106 raft_consensus.cc:3037] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.728770 8246 raft_consensus.cc:3037] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.729398 8106 raft_consensus.cc:2454] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 1.
I20260925 12:12:33.729529 8246 raft_consensus.cc:2454] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 1.
I20260925 12:12:33.729573 8317 leader_election.cc:302] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0896f644a79546128dcf15119d04e1c5, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:33.729660 8466 raft_consensus.cc:2781] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:33.729812 8466 raft_consensus.cc:686] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Running, Role: LEADER
I20260925 12:12:33.729920 8466 consensus_queue.cc:237] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260925 12:12:33.730755 7949 catalog_manager.cc:5495] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0896f644a79546128dcf15119d04e1c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0896f644a79546128dcf15119d04e1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20260925 12:12:33.736716 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.734351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.0.98.3:42739)
0925 12:12:33.734406 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.734408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.734415 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:33.734441 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.734442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.734461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.734537 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:33.734686 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.735151 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.735904 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.736114 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.736197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.736375 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:33.736474 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:33.736582 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.736623 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.736642 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:33.738334 8106 raft_consensus.cc:1261] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Refusing update from remote peer 0896f644a79546128dcf15119d04e1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:33.738334 8246 raft_consensus.cc:1261] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Refusing update from remote peer 0896f644a79546128dcf15119d04e1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:33.738690 8470 consensus_queue.cc:1035] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:33.738798 8466 consensus_queue.cc:1035] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:33.741120 8474 mvcc.cc:205] Tried to move back new op lower bound from 7333225896909717504 to 7333225896878624768. Current Snapshot: MvccSnapshot[applied={T|T < 7333225896909717504}]
I20260925 12:12:33.743147 7949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:33.743253 7949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:33.743935 7949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Sending DeleteTablet for 3 replicas of tablet 3a23af98b2cf49a1936ea1c33644cf2c
I20260925 12:12:33.744248 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58486
I20260925 12:12:33.744254 8366 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59686
I20260925 12:12:33.744354 8226 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a23af98b2cf49a1936ea1c33644cf2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42212
I20260925 12:12:33.744585 8483 tablet_replica.cc:330] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: stopping tablet replica
I20260925 12:12:33.744582 8484 tablet_replica.cc:330] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: stopping tablet replica
I20260925 12:12:33.744679 8483 raft_consensus.cc:2229] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:33.744664 7949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 12:12:33.744771 8483 raft_consensus.cc:2258] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 12:12:33.744807 7949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:33.744951 8484 raft_consensus.cc:2229] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:33.745114 8484 raft_consensus.cc:2258] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:33.745226 8483 ts_tablet_manager.cc:1905] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:33.745946 8086 tablet_service.cc:1467] Processing CreateTablet for tablet 6af9ea634c9745c08756786359e29d59 (DEFAULT_TABLE table=test-table [id=ee733416b36943d9b6b05574b229ad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.745971 8226 tablet_service.cc:1467] Processing CreateTablet for tablet 6af9ea634c9745c08756786359e29d59 (DEFAULT_TABLE table=test-table [id=ee733416b36943d9b6b05574b229ad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.745987 8366 tablet_service.cc:1467] Processing CreateTablet for tablet 6af9ea634c9745c08756786359e29d59 (DEFAULT_TABLE table=test-table [id=ee733416b36943d9b6b05574b229ad50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:33.746068 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af9ea634c9745c08756786359e29d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.746083 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af9ea634c9745c08756786359e29d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.746083 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af9ea634c9745c08756786359e29d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.747292 8462 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Bootstrap starting.
I20260925 12:12:33.747438 8460 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Bootstrap starting.
I20260925 12:12:33.747718 8462 tablet_bootstrap.cc:654] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.748090 8460 tablet_bootstrap.cc:654] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.748191 8482 tablet_replica.cc:330] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: stopping tablet replica
I20260925 12:12:33.748313 8482 raft_consensus.cc:2229] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:33.748394 8461 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Bootstrap starting.
I20260925 12:12:33.748477 8482 raft_consensus.cc:2258] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:33.748782 8461 tablet_bootstrap.cc:654] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.749126 8482 ts_tablet_manager.cc:1905] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:33.749691 8462 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: No bootstrap required, opened a new log
I20260925 12:12:33.749740 8462 ts_tablet_manager.cc:1397] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:33.749879 8462 raft_consensus.cc:348] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.749959 8462 raft_consensus.cc:374] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.749986 8462 raft_consensus.cc:729] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.750034 8462 consensus_queue.cc:260] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.750133 8462 ts_tablet_manager.cc:1428] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.750614 8483 ts_tablet_manager.cc:1918] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:33.750650 8482 ts_tablet_manager.cc:1918] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:33.750689 8483 log.cc:1196] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/3a23af98b2cf49a1936ea1c33644cf2c
I20260925 12:12:33.750720 8482 log.cc:1196] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/3a23af98b2cf49a1936ea1c33644cf2c
I20260925 12:12:33.750941 8483 ts_tablet_manager.cc:1939] T 3a23af98b2cf49a1936ea1c33644cf2c P 0896f644a79546128dcf15119d04e1c5: Deleting consensus metadata
I20260925 12:12:33.751004 8482 ts_tablet_manager.cc:1939] T 3a23af98b2cf49a1936ea1c33644cf2c P a9a7be24d9754b89b49b71e9783aa9ca: Deleting consensus metadata
I20260925 12:12:33.751362 7936 catalog_manager.cc:4841] TS 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739): tablet 3a23af98b2cf49a1936ea1c33644cf2c (table test-table [id=8dac6484694a4287b643311df016e0cb]) successfully deleted
I20260925 12:12:33.751394 7933 catalog_manager.cc:4841] TS a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411): tablet 3a23af98b2cf49a1936ea1c33644cf2c (table test-table [id=8dac6484694a4287b643311df016e0cb]) successfully deleted
I20260925 12:12:33.751610 8461 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: No bootstrap required, opened a new log
I20260925 12:12:33.751618 8460 tablet_bootstrap.cc:492] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: No bootstrap required, opened a new log
I20260925 12:12:33.751654 8461 ts_tablet_manager.cc:1397] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 12:12:33.751657 8460 ts_tablet_manager.cc:1397] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 12:12:33.751786 8460 raft_consensus.cc:348] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.751796 8461 raft_consensus.cc:348] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.751878 8460 raft_consensus.cc:374] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.751879 8461 raft_consensus.cc:374] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.751901 8460 raft_consensus.cc:729] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.751905 8461 raft_consensus.cc:729] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.751940 8460 consensus_queue.cc:260] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.751967 8461 consensus_queue.cc:260] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.752024 8460 ts_tablet_manager.cc:1428] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:33.752059 8461 ts_tablet_manager.cc:1428] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:33.756718 8484 ts_tablet_manager.cc:1905] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:33.758137 8484 ts_tablet_manager.cc:1918] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:33.758232 8484 log.cc:1196] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/3a23af98b2cf49a1936ea1c33644cf2c
I20260925 12:12:33.758450 8484 ts_tablet_manager.cc:1939] T 3a23af98b2cf49a1936ea1c33644cf2c P 086e23ca752a4a11975c79ed7076286e: Deleting consensus metadata
I20260925 12:12:33.758832 7936 catalog_manager.cc:4841] TS 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831): tablet 3a23af98b2cf49a1936ea1c33644cf2c (table test-table [id=8dac6484694a4287b643311df016e0cb]) successfully deleted
W20260925 12:12:33.770320 8153 tablet.cc:2369] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:12:33.801568 8467 raft_consensus.cc:482] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:33.801680 8467 raft_consensus.cc:504] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.802032 8467 leader_election.cc:288] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:33.805459 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.802210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.98.2:55015)
0925 12:12:33.802318 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.802329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.802373 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.802626 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.802627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.802629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.802710 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.802712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.803572 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.803573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.804659 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.804660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.805180 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.805181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.805256 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.805258 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.805332 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.805379 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 12:12:33.805450 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.802466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55015 (local address 127.0.98.1:42411)
0925 12:12:33.802518 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.802520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.802529 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:33.802552 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.802553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.802561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.802608 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.802801 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.803482 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.804744 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.805094 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.805182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.805193 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.805342 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.805378 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:12:33.805691 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.802267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50365 (local address 127.0.98.3:42739)
0925 12:12:33.802408 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.802410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.802445 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:12:33.802476 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.802477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.802485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.802546 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.802725 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.803208 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.804726 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.805278 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.805352 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.805363 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.805582 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.805621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260925 12:12:33.805934 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.802218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42739 (local address 127.0.98.2:50365)
0925 12:12:33.802360 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.802371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.802424 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.802553 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.802555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.802557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.802646 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.802648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.803301 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.803302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.804633 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.804637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.805416 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.805418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.805489 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.805493 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.805785 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.805830 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":13,"thread_start_us":59,"threads_started":1}
I20260925 12:12:33.806042 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af9ea634c9745c08756786359e29d59" candidate_uuid: "086e23ca752a4a11975c79ed7076286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" is_pre_election: true
I20260925 12:12:33.806170 8106 raft_consensus.cc:2454] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086e23ca752a4a11975c79ed7076286e in term 0.
I20260925 12:12:33.806247 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af9ea634c9745c08756786359e29d59" candidate_uuid: "086e23ca752a4a11975c79ed7076286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5" is_pre_election: true
I20260925 12:12:33.806372 8386 raft_consensus.cc:2454] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086e23ca752a4a11975c79ed7076286e in term 0.
I20260925 12:12:33.806386 8177 leader_election.cc:302] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:33.806515 8467 raft_consensus.cc:2781] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:33.806557 8467 raft_consensus.cc:482] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:33.806581 8467 raft_consensus.cc:3037] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.807221 8467 raft_consensus.cc:504] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.807365 8467 leader_election.cc:288] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [CANDIDATE]: Term 1 election: Requested vote from peers a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:33.807539 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af9ea634c9745c08756786359e29d59" candidate_uuid: "086e23ca752a4a11975c79ed7076286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
I20260925 12:12:33.807536 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af9ea634c9745c08756786359e29d59" candidate_uuid: "086e23ca752a4a11975c79ed7076286e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5"
I20260925 12:12:33.807610 8106 raft_consensus.cc:3037] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.807610 8386 raft_consensus.cc:3037] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.808215 8106 raft_consensus.cc:2454] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086e23ca752a4a11975c79ed7076286e in term 1.
I20260925 12:12:33.808357 8177 leader_election.cc:302] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:33.808369 8386 raft_consensus.cc:2454] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086e23ca752a4a11975c79ed7076286e in term 1.
I20260925 12:12:33.808449 8467 raft_consensus.cc:2781] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:33.808612 8467 raft_consensus.cc:686] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 1 LEADER]: Becoming Leader. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Running, Role: LEADER
I20260925 12:12:33.808699 8467 consensus_queue.cc:237] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.809412 7948 catalog_manager.cc:5495] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e reported cstate change: term changed from 0 to 1, leader changed from <none> to 086e23ca752a4a11975c79ed7076286e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "086e23ca752a4a11975c79ed7076286e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } health_report { overall_health: UNKNOWN } } }
W20260925 12:12:33.820626 8433 tablet.cc:2369] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 12:12:33 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338353, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:33.852977 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.845284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.0.98.62:35593)
0925 12:12:33.845335 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.845337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.845471 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:12:33.845541 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.845542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.845550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.845601 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:33.846177 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.846805 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.847829 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.848013 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.848065 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.850298 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:33.850299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:33.850299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:33.850305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:33.851782 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.852212 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.852213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.852214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.852269 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:33.852457 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:33.852458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:33.852458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:33.852611 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:33.852663 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.852842 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.852908 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:12:33.853116 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.845182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:41334)
0925 12:12:33.845373 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.845384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.845478 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.845632 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.845633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.846038 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:33.846097 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.846100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.846889 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.846890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.847739 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.847742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.848094 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.848095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.848163 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.848718 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:33.848723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:33.850218 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:33.851859 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.851867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.851869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.852115 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.852330 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:33.852331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:33.852331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:33.852397 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:33.852688 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:33.852694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:33.852743 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.852824 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.852977 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":43,"thread_start_us":99,"threads_started":1}
I20260925 12:12:33.855026 7948 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41334:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 12:12:33.855242 7948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:33.856668 8086 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7e8a26d2846059746aba1e881a0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e1d7fbc2b14b3392d56ffee72aaf46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:12:33.856719 8226 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7e8a26d2846059746aba1e881a0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e1d7fbc2b14b3392d56ffee72aaf46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:12:33.856827 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7e8a26d2846059746aba1e881a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.856796 8366 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7e8a26d2846059746aba1e881a0b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e1d7fbc2b14b3392d56ffee72aaf46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 12:12:33.856879 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7e8a26d2846059746aba1e881a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.856923 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7e8a26d2846059746aba1e881a0b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:33.858299 8460 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: Bootstrap starting.
I20260925 12:12:33.858318 8462 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: Bootstrap starting.
I20260925 12:12:33.858650 8462 tablet_bootstrap.cc:654] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.858675 8460 tablet_bootstrap.cc:654] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.858984 8461 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: Bootstrap starting.
I20260925 12:12:33.859290 8460 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: No bootstrap required, opened a new log
I20260925 12:12:33.859345 8460 ts_tablet_manager.cc:1397] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:33.859382 8461 tablet_bootstrap.cc:654] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:33.859495 8460 raft_consensus.cc:348] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.859578 8460 raft_consensus.cc:374] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.859598 8460 raft_consensus.cc:729] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.859645 8460 consensus_queue.cc:260] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.859746 8460 ts_tablet_manager.cc:1428] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:33.859862 8468 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:33.859931 8468 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:33.860549 8461 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: No bootstrap required, opened a new log
I20260925 12:12:33.860599 8461 ts_tablet_manager.cc:1397] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:12:33.860729 8461 raft_consensus.cc:348] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.860776 8461 raft_consensus.cc:374] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.860790 8461 raft_consensus.cc:729] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.860823 8461 consensus_queue.cc:260] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.860914 8461 ts_tablet_manager.cc:1428] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.860973 8467 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:33.861042 8467 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:33.861668 8462 tablet_bootstrap.cc:492] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: No bootstrap required, opened a new log
I20260925 12:12:33.861743 8462 ts_tablet_manager.cc:1397] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 12:12:33.861858 8462 raft_consensus.cc:348] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.861943 8462 raft_consensus.cc:374] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:33.861959 8462 raft_consensus.cc:729] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Initialized, Role: FOLLOWER
I20260925 12:12:33.862028 8462 consensus_queue.cc:260] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.862118 8462 ts_tablet_manager.cc:1428] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:33.862131 8466 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:33.862197 8466 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:33.976191 8468 raft_consensus.cc:482] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:33.976289 8468 raft_consensus.cc:504] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.976655 8468 leader_election.cc:288] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:33.979707 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.977094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56477 (local address 127.0.98.3:42739)
0925 12:12:33.977141 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.977144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.977202 (+ 58us) server_negotiation.cc:407] Connection header received
0925 12:12:33.977270 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.977271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.977283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.977328 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.977468 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.977961 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.978975 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.979335 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.979429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.979439 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.979595 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.979634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:12:33.979708 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.976948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53181 (local address 127.0.98.2:37831)
0925 12:12:33.977044 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:33.977046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:33.977113 (+ 67us) server_negotiation.cc:407] Connection header received
0925 12:12:33.977189 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:33.977190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:33.977200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:33.977272 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:33.977447 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.977935 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.978997 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.979339 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.979429 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.979438 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:33.979595 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 12:12:33.979636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 12:12:33.979738 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.976831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37831 (local address 127.0.98.1:53181)
0925 12:12:33.977046 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.977058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.977134 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.977276 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.977277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.977279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.977368 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.977370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.978017 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.978018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.978889 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.978893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.979433 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.979434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.979511 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.979514 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.979587 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.979633 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":56,"thread_start_us":94,"threads_started":1}
I20260925 12:12:33.980002 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7e8a26d2846059746aba1e881a0b2" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e" is_pre_election: true
I20260925 12:12:33.980091 8246 raft_consensus.cc:2454] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 0.
I20260925 12:12:33.979745 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:33.977037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42739 (local address 127.0.98.1:56477)
0925 12:12:33.977151 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:33.977161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:33.977219 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:33.977337 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:33.977338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:33.977340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:33.977401 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.977402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.978041 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.978042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.978877 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:33.978879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:33.979431 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:33.979432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:33.979502 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:33.979506 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:33.979587 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:12:33.979633 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":14,"thread_start_us":46,"threads_started":1}
I20260925 12:12:33.980274 8040 leader_election.cc:302] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:33.980415 8468 raft_consensus.cc:2781] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:33.980470 8468 raft_consensus.cc:482] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:33.980516 8468 raft_consensus.cc:3037] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.980574 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7e8a26d2846059746aba1e881a0b2" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5" is_pre_election: true
I20260925 12:12:33.980656 8386 raft_consensus.cc:2454] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 0.
I20260925 12:12:33.981274 8468 raft_consensus.cc:504] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.981370 8468 leader_election.cc:288] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 election: Requested vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:33.981555 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7e8a26d2846059746aba1e881a0b2" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e"
I20260925 12:12:33.981585 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7e8a26d2846059746aba1e881a0b2" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5"
I20260925 12:12:33.981642 8246 raft_consensus.cc:3037] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.981647 8386 raft_consensus.cc:3037] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:33.982426 8386 raft_consensus.cc:2454] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 1.
I20260925 12:12:33.982422 8246 raft_consensus.cc:2454] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 1.
I20260925 12:12:33.982591 8040 leader_election.cc:302] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:33.982689 8468 raft_consensus.cc:2781] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:33.982811 8468 raft_consensus.cc:686] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 LEADER]: Becoming Leader. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Running, Role: LEADER
I20260925 12:12:33.982883 8468 consensus_queue.cc:237] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:33.983412 8502 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: TxnStatusTablet state changed. Reason: New leader a9a7be24d9754b89b49b71e9783aa9ca. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:33.983516 8502 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:12:33.983690 8504 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 12:12:33.983649 7949 catalog_manager.cc:5495] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca reported cstate change: term changed from 0 to 1, leader changed from <none> to a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } health_report { overall_health: UNKNOWN } } }
I20260925 12:12:33.983795 8504 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 12:12:34.009909 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.007288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.0.98.1:42411)
0925 12:12:34.007335 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.007337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.007437 (+ 100us) server_negotiation.cc:407] Connection header received
0925 12:12:34.007534 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.007535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.007554 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.007635 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.007771 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.008262 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.009091 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.009296 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.009381 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.009574 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.009674 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.009756 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.009809 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.009828 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:12:34.009927 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.007220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.0.1:58500)
0925 12:12:34.007357 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:34.007368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:34.007448 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:34.007642 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:34.007642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:34.007644 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:12:34.007693 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.007695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.008349 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.008350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.009005 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.009008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.009382 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.009382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.009448 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.009463 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.009740 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.009748 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 12:12:34.009803 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:12:34.009849 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 12:12:34.011247 8246 raft_consensus.cc:1261] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Refusing update from remote peer a9a7be24d9754b89b49b71e9783aa9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.011523 8386 raft_consensus.cc:1261] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Refusing update from remote peer a9a7be24d9754b89b49b71e9783aa9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.011687 8468 consensus_queue.cc:1035] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.011840 8502 consensus_queue.cc:1035] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.012251 8467 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: TxnStatusTablet state changed. Reason: New leader a9a7be24d9754b89b49b71e9783aa9ca. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.012341 8467 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:34.012523 8466 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: TxnStatusTablet state changed. Reason: New leader a9a7be24d9754b89b49b71e9783aa9ca. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.012612 8466 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:34.013360 8468 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.013459 8468 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:12:34.013652 8468 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.013717 8468 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P a9a7be24d9754b89b49b71e9783aa9ca: This TxnStatusTablet replica's current role is: LEADER
I20260925 12:12:34.014038 8473 mvcc.cc:205] Tried to move back new op lower bound from 7333225898027388928 to 7333225897914613760. Current Snapshot: MvccSnapshot[applied={T|T < 7333225898027388928}]
I20260925 12:12:34.014159 8467 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.014227 8467 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 086e23ca752a4a11975c79ed7076286e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:34.014364 8466 tablet_replica.cc:439] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } } }
I20260925 12:12:34.014428 8466 tablet_replica.cc:442] T 8cd7e8a26d2846059746aba1e881a0b2 P 0896f644a79546128dcf15119d04e1c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 12:12:34.017637 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.015222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.0.98.2:37831)
0925 12:12:34.015276 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.015278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.015285 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:34.015318 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.015319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.015342 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.015413 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.015551 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.016011 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.016818 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.017053 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.017139 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.017297 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.017403 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.017486 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.017526 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.017544 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:34.022538 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.019879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38141 (local address 127.0.98.1:42411)
0925 12:12:34.019939 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.019941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.020056 (+ 115us) server_negotiation.cc:407] Connection header received
0925 12:12:34.020117 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.020118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.020127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.020178 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.020329 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.020819 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.021961 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.022132 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.022182 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.022359 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.022382 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.022421 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.022475 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.022491 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:12:34.022604 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.019811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.98.2:38141)
0925 12:12:34.019992 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:34.020001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:34.020068 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:34.020204 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:34.020205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:34.020207 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:12:34.020263 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.020264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.020927 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.020928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.021884 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.021886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.022202 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.022203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.022267 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.022281 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.022422 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.022424 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:34.022474 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:34.022513 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260925 12:12:34.025067 8386 raft_consensus.cc:1261] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Refusing update from remote peer 086e23ca752a4a11975c79ed7076286e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.025102 8106 raft_consensus.cc:1261] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Refusing update from remote peer 086e23ca752a4a11975c79ed7076286e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.025285 8467 consensus_queue.cc:1035] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.025416 8489 consensus_queue.cc:1035] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.026993 8475 mvcc.cc:205] Tried to move back new op lower bound from 7333225898085031936 to 7333225897201139712. Current Snapshot: MvccSnapshot[applied={T|T < 7333225898085031936}]
I20260925 12:12:34.034250 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.031290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34045 (local address 127.0.98.2:37831)
0925 12:12:34.031337 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.031345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.031354 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:34.031390 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.031391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.031401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.031457 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.031553 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.032285 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.033392 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.033712 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.033775 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.033981 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.034032 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.034086 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.034169 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.034188 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:12:34.034267 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.031111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37831 (local address 127.0.98.1:34045)
0925 12:12:34.031271 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:34.031284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:34.031333 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:34.031458 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:34.031459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:34.031460 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:12:34.031513 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.031515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.032366 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.032367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.033319 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.033321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.033809 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.033810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.033863 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.033876 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.034113 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.034116 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:34.034162 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:12:34.034204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260925 12:12:34.042426 7948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.042537 7948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.043123 7948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Sending DeleteTablet for 3 replicas of tablet 6af9ea634c9745c08756786359e29d59
I20260925 12:12:34.043387 8226 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af9ea634c9745c08756786359e29d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42212
I20260925 12:12:34.043387 8366 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af9ea634c9745c08756786359e29d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59686
I20260925 12:12:34.043500 8484 tablet_replica.cc:330] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: stopping tablet replica
I20260925 12:12:34.043507 8483 tablet_replica.cc:330] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: stopping tablet replica
I20260925 12:12:34.043552 8484 raft_consensus.cc:2229] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:34.043576 8483 raft_consensus.cc:2229] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.043612 8483 raft_consensus.cc:2258] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.043634 8484 raft_consensus.cc:2258] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.043954 8483 ts_tablet_manager.cc:1905] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.043987 8484 ts_tablet_manager.cc:1905] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.046772 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af9ea634c9745c08756786359e29d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58486
I20260925 12:12:34.046864 8482 tablet_replica.cc:330] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: stopping tablet replica
I20260925 12:12:34.046851 8483 ts_tablet_manager.cc:1918] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:12:34.046900 8483 log.cc:1196] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/6af9ea634c9745c08756786359e29d59
I20260925 12:12:34.046918 8482 raft_consensus.cc:2229] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.046954 8482 raft_consensus.cc:2258] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.047114 8483 ts_tablet_manager.cc:1939] T 6af9ea634c9745c08756786359e29d59 P 0896f644a79546128dcf15119d04e1c5: Deleting consensus metadata
I20260925 12:12:34.047401 8484 ts_tablet_manager.cc:1918] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:12:34.047417 7936 catalog_manager.cc:4841] TS 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739): tablet 6af9ea634c9745c08756786359e29d59 (table test-table [id=ee733416b36943d9b6b05574b229ad50]) successfully deleted
I20260925 12:12:34.047458 8484 log.cc:1196] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/6af9ea634c9745c08756786359e29d59
I20260925 12:12:34.047649 8484 ts_tablet_manager.cc:1939] T 6af9ea634c9745c08756786359e29d59 P 086e23ca752a4a11975c79ed7076286e: Deleting consensus metadata
I20260925 12:12:34.047901 7936 catalog_manager.cc:4841] TS 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831): tablet 6af9ea634c9745c08756786359e29d59 (table test-table [id=ee733416b36943d9b6b05574b229ad50]) successfully deleted
I20260925 12:12:34.048779 8482 ts_tablet_manager.cc:1905] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.050427 8482 ts_tablet_manager.cc:1918] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:12:34.050506 8482 log.cc:1196] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/6af9ea634c9745c08756786359e29d59
I20260925 12:12:34.050702 8482 ts_tablet_manager.cc:1939] T 6af9ea634c9745c08756786359e29d59 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting consensus metadata
I20260925 12:12:34.051007 7933 catalog_manager.cc:4841] TS a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411): tablet 6af9ea634c9745c08756786359e29d59 (table test-table [id=ee733416b36943d9b6b05574b229ad50]) successfully deleted
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[7904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:12:34.077000 7998 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:34.066705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.0.98.62:35593)
0925 12:12:34.066751 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.066753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.067101 (+ 348us) server_negotiation.cc:407] Connection header received
0925 12:12:34.067200 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.067201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.067211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.067238 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:34.068002 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.068665 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.069419 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.069609 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.069685 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.076945 (+ 7260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41336: BlockingRecv error: recv got EOF from 127.0.0.1:41336 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[7904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 12:12:34.088737 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.078141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.98.62:35593)
0925 12:12:34.078199 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.078201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.081762 (+ 3561us) server_negotiation.cc:407] Connection header received
0925 12:12:34.081843 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.081844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.081855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.081928 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:34.082421 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.082902 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.083605 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.083796 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.083868 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.086075 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:34.086077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:34.086077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:34.086085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:34.087506 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:34.087853 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:34.087854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:34.087854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:34.087929 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:34.088142 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:34.088142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:34.088143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:34.088276 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:34.088377 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.088542 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.088616 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:12:34.090186 7948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:12:34.090364 7948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:34.091629 8086 tablet_service.cc:1467] Processing CreateTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 (DEFAULT_TABLE table=test-table [id=29eeb68eb8884409875c0afa6aaacf53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.091763 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff88931d2644f0aaf8dc7a3ca92ca76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.091749 8226 tablet_service.cc:1467] Processing CreateTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 (DEFAULT_TABLE table=test-table [id=29eeb68eb8884409875c0afa6aaacf53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.091850 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff88931d2644f0aaf8dc7a3ca92ca76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.092584 8366 tablet_service.cc:1467] Processing CreateTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 (DEFAULT_TABLE table=test-table [id=29eeb68eb8884409875c0afa6aaacf53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.092695 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff88931d2644f0aaf8dc7a3ca92ca76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.093154 8460 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Bootstrap starting.
I20260925 12:12:34.093552 8460 tablet_bootstrap.cc:654] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.094046 8462 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Bootstrap starting.
I20260925 12:12:34.094399 8462 tablet_bootstrap.cc:654] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.095263 8461 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Bootstrap starting.
I20260925 12:12:34.095533 8462 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: No bootstrap required, opened a new log
I20260925 12:12:34.095573 8462 ts_tablet_manager.cc:1397] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:34.095660 8461 tablet_bootstrap.cc:654] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.095702 8462 raft_consensus.cc:348] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.095780 8462 raft_consensus.cc:374] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.095801 8462 raft_consensus.cc:729] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.095849 8462 consensus_queue.cc:260] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.095944 8462 ts_tablet_manager.cc:1428] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:34.096645 8460 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: No bootstrap required, opened a new log
I20260925 12:12:34.096689 8460 ts_tablet_manager.cc:1397] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 12:12:34.096748 8461 tablet_bootstrap.cc:492] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: No bootstrap required, opened a new log
I20260925 12:12:34.096787 8461 ts_tablet_manager.cc:1397] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:34.096840 8460 raft_consensus.cc:348] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.096922 8460 raft_consensus.cc:374] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.096915 8461 raft_consensus.cc:348] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.096969 8460 raft_consensus.cc:729] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.096978 8461 raft_consensus.cc:374] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.097002 8461 raft_consensus.cc:729] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.097013 8460 consensus_queue.cc:260] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.097033 8461 consensus_queue.cc:260] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.097107 8460 ts_tablet_manager.cc:1428] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:34.097118 8461 ts_tablet_manager.cc:1428] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:34.149684 8466 raft_consensus.cc:482] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:34.149794 8466 raft_consensus.cc:504] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.149955 8466 leader_election.cc:288] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411), 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831)
I20260925 12:12:34.150161 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff88931d2644f0aaf8dc7a3ca92ca76" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" is_pre_election: true
I20260925 12:12:34.150254 8106 raft_consensus.cc:2454] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 0.
I20260925 12:12:34.150220 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff88931d2644f0aaf8dc7a3ca92ca76" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e" is_pre_election: true
I20260925 12:12:34.150295 8246 raft_consensus.cc:2454] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 0.
I20260925 12:12:34.150414 8317 leader_election.cc:302] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0896f644a79546128dcf15119d04e1c5, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:34.150513 8466 raft_consensus.cc:2781] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:34.150542 8466 raft_consensus.cc:482] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:34.150559 8466 raft_consensus.cc:3037] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.151319 8466 raft_consensus.cc:504] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.151451 8466 leader_election.cc:288] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 election: Requested vote from peers a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411), 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831)
I20260925 12:12:34.151621 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff88931d2644f0aaf8dc7a3ca92ca76" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a7be24d9754b89b49b71e9783aa9ca"
I20260925 12:12:34.151635 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff88931d2644f0aaf8dc7a3ca92ca76" candidate_uuid: "0896f644a79546128dcf15119d04e1c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e"
I20260925 12:12:34.151686 8106 raft_consensus.cc:3037] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.151690 8246 raft_consensus.cc:3037] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.152431 8246 raft_consensus.cc:2454] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 1.
I20260925 12:12:34.152431 8106 raft_consensus.cc:2454] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896f644a79546128dcf15119d04e1c5 in term 1.
I20260925 12:12:34.152621 8320 leader_election.cc:302] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, 0896f644a79546128dcf15119d04e1c5; no voters:
I20260925 12:12:34.152799 8466 raft_consensus.cc:2781] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:34.152886 8466 raft_consensus.cc:686] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Running, Role: LEADER
I20260925 12:12:34.152946 8466 consensus_queue.cc:237] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.153548 7949 catalog_manager.cc:5495] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0896f644a79546128dcf15119d04e1c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0896f644a79546128dcf15119d04e1c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } health_report { overall_health: HEALTHY } } }
I20260925 12:12:34.172971 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.170553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.0.98.3:42739)
0925 12:12:34.170614 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.170616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.170626 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:34.170687 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.170689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.170700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.170745 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.170884 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.171365 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.172099 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.172392 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.172453 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.172639 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.172673 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.172724 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.172896 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.172915 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:34.173777 8246 raft_consensus.cc:1261] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Refusing update from remote peer 0896f644a79546128dcf15119d04e1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.173779 8106 raft_consensus.cc:1261] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Refusing update from remote peer 0896f644a79546128dcf15119d04e1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.174299 8470 consensus_queue.cc:1035] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.174446 8466 consensus_queue.cc:1035] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.177254 7947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.177361 7947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.178053 7947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Sending DeleteTablet for 3 replicas of tablet cff88931d2644f0aaf8dc7a3ca92ca76
I20260925 12:12:34.178323 8226 tablet_service.cc:1514] Processing DeleteTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42212
I20260925 12:12:34.178333 8366 tablet_service.cc:1514] Processing DeleteTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59686
I20260925 12:12:34.178375 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet cff88931d2644f0aaf8dc7a3ca92ca76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58486
I20260925 12:12:34.178412 8483 tablet_replica.cc:330] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: stopping tablet replica
I20260925 12:12:34.178418 8484 tablet_replica.cc:330] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: stopping tablet replica
I20260925 12:12:34.178457 8482 tablet_replica.cc:330] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: stopping tablet replica
I20260925 12:12:34.178459 8483 raft_consensus.cc:2229] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:34.178463 8484 raft_consensus.cc:2229] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.178496 8482 raft_consensus.cc:2229] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.178499 8484 raft_consensus.cc:2258] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.178508 8483 raft_consensus.cc:2258] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.178536 8482 raft_consensus.cc:2258] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.178843 8484 ts_tablet_manager.cc:1905] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.178937 8483 ts_tablet_manager.cc:1905] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.178937 8482 ts_tablet_manager.cc:1905] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.180604 7947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:12:34.180755 7947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:34.180943 8484 ts_tablet_manager.cc:1918] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:34.180967 8483 ts_tablet_manager.cc:1918] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:34.180967 8482 ts_tablet_manager.cc:1918] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:34.181005 8484 log.cc:1196] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/cff88931d2644f0aaf8dc7a3ca92ca76
I20260925 12:12:34.181016 8483 log.cc:1196] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/cff88931d2644f0aaf8dc7a3ca92ca76
I20260925 12:12:34.181016 8482 log.cc:1196] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/cff88931d2644f0aaf8dc7a3ca92ca76
I20260925 12:12:34.181231 8484 ts_tablet_manager.cc:1939] T cff88931d2644f0aaf8dc7a3ca92ca76 P 086e23ca752a4a11975c79ed7076286e: Deleting consensus metadata
I20260925 12:12:34.181241 8482 ts_tablet_manager.cc:1939] T cff88931d2644f0aaf8dc7a3ca92ca76 P a9a7be24d9754b89b49b71e9783aa9ca: Deleting consensus metadata
I20260925 12:12:34.181241 8483 ts_tablet_manager.cc:1939] T cff88931d2644f0aaf8dc7a3ca92ca76 P 0896f644a79546128dcf15119d04e1c5: Deleting consensus metadata
I20260925 12:12:34.181525 7936 catalog_manager.cc:4841] TS 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739): tablet cff88931d2644f0aaf8dc7a3ca92ca76 (table test-table [id=29eeb68eb8884409875c0afa6aaacf53]) successfully deleted
I20260925 12:12:34.181587 7936 catalog_manager.cc:4841] TS 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831): tablet cff88931d2644f0aaf8dc7a3ca92ca76 (table test-table [id=29eeb68eb8884409875c0afa6aaacf53]) successfully deleted
I20260925 12:12:34.181721 7933 catalog_manager.cc:4841] TS a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411): tablet cff88931d2644f0aaf8dc7a3ca92ca76 (table test-table [id=29eeb68eb8884409875c0afa6aaacf53]) successfully deleted
I20260925 12:12:34.182560 8086 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a (DEFAULT_TABLE table=test-table [id=30f4596449544684a4015d547ba0af6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.182667 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf128bf2364d89a51a2c5bea408d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.182684 8366 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a (DEFAULT_TABLE table=test-table [id=30f4596449544684a4015d547ba0af6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.182761 8366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf128bf2364d89a51a2c5bea408d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.183851 8460 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: Bootstrap starting.
I20260925 12:12:34.183887 8226 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a (DEFAULT_TABLE table=test-table [id=30f4596449544684a4015d547ba0af6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:34.183995 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf128bf2364d89a51a2c5bea408d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:34.184154 8460 tablet_bootstrap.cc:654] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.184366 8462 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Bootstrap starting.
I20260925 12:12:34.184674 8462 tablet_bootstrap.cc:654] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.185043 8460 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: No bootstrap required, opened a new log
I20260925 12:12:34.185089 8460 ts_tablet_manager.cc:1397] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:34.185199 8460 raft_consensus.cc:348] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.185271 8460 raft_consensus.cc:374] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.185266 8461 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Bootstrap starting.
I20260925 12:12:34.185294 8460 raft_consensus.cc:729] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.185338 8460 consensus_queue.cc:260] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.185436 8460 ts_tablet_manager.cc:1428] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:34.185642 8461 tablet_bootstrap.cc:654] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:34.186244 8461 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: No bootstrap required, opened a new log
I20260925 12:12:34.186275 8462 tablet_bootstrap.cc:492] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: No bootstrap required, opened a new log
I20260925 12:12:34.186292 8461 ts_tablet_manager.cc:1397] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 12:12:34.186311 8462 ts_tablet_manager.cc:1397] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:12:34.186408 8462 raft_consensus.cc:348] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.186468 8462 raft_consensus.cc:374] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.186434 8461 raft_consensus.cc:348] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.186489 8462 raft_consensus.cc:729] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896f644a79546128dcf15119d04e1c5, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.186493 8461 raft_consensus.cc:374] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:34.186515 8461 raft_consensus.cc:729] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e23ca752a4a11975c79ed7076286e, State: Initialized, Role: FOLLOWER
I20260925 12:12:34.186538 8462 consensus_queue.cc:260] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.186553 8461 consensus_queue.cc:260] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.186627 8462 ts_tablet_manager.cc:1428] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:34.186631 8461 ts_tablet_manager.cc:1428] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:34.216763 8468 raft_consensus.cc:482] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:34.216883 8468 raft_consensus.cc:504] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.217060 8468 leader_election.cc:288] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:34.217274 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf128bf2364d89a51a2c5bea408d9a" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e" is_pre_election: true
I20260925 12:12:34.217306 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf128bf2364d89a51a2c5bea408d9a" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5" is_pre_election: true
I20260925 12:12:34.217399 8246 raft_consensus.cc:2454] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 0.
I20260925 12:12:34.217417 8386 raft_consensus.cc:2454] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 0.
I20260925 12:12:34.217583 8040 leader_election.cc:302] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086e23ca752a4a11975c79ed7076286e, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:34.217715 8468 raft_consensus.cc:2781] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:34.217787 8468 raft_consensus.cc:482] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:34.217818 8468 raft_consensus.cc:3037] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.218545 8468 raft_consensus.cc:504] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.218691 8468 leader_election.cc:288] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 election: Requested vote from peers 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831), 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739)
I20260925 12:12:34.219004 8386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf128bf2364d89a51a2c5bea408d9a" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896f644a79546128dcf15119d04e1c5"
I20260925 12:12:34.219004 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf128bf2364d89a51a2c5bea408d9a" candidate_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086e23ca752a4a11975c79ed7076286e"
I20260925 12:12:34.219094 8246 raft_consensus.cc:3037] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.219094 8386 raft_consensus.cc:3037] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:34.219836 8386 raft_consensus.cc:2454] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 1.
I20260925 12:12:34.219906 8246 raft_consensus.cc:2454] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9a7be24d9754b89b49b71e9783aa9ca in term 1.
I20260925 12:12:34.220060 8040 leader_election.cc:302] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0896f644a79546128dcf15119d04e1c5, a9a7be24d9754b89b49b71e9783aa9ca; no voters:
I20260925 12:12:34.220170 8468 raft_consensus.cc:2781] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:34.220258 8468 raft_consensus.cc:686] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 1 LEADER]: Becoming Leader. State: Replica: a9a7be24d9754b89b49b71e9783aa9ca, State: Running, Role: LEADER
I20260925 12:12:34.220362 8468 consensus_queue.cc:237] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } }
I20260925 12:12:34.220975 7949 catalog_manager.cc:5495] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca reported cstate change: term changed from 0 to 1, leader changed from <none> to a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9a7be24d9754b89b49b71e9783aa9ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 } health_report { overall_health: UNKNOWN } } }
I20260925 12:12:34.236858 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.234687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.98.1:42411)
0925 12:12:34.234739 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.234741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.234748 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:34.234779 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.234780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.234796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.234851 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.234964 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.235436 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.236159 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.236362 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.236436 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.236594 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.236625 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.236690 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.236760 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.236779 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:12:34.240731 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.238917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41939 (local address 127.0.98.1:42411)
0925 12:12:34.238971 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:34.238974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:34.238981 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:34.238997 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:34.238997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:34.239004 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:34.239050 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:34.239147 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.239573 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.240197 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.240359 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.240404 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.240558 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.240579 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.240618 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:34.240662 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 12:12:34.240675 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:34.240739 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.238832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.98.1:41939)
0925 12:12:34.238887 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:34.238898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:34.238941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:34.239064 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:34.239065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:34.239066 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 12:12:34.239095 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.239096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.239628 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.239629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.240135 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:34.240136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:34.240415 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:34.240415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:34.240451 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:34.240455 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 12:12:34.240624 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 12:12:34.240625 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 12:12:34.240653 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 12:12:34.240682 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:12:34.242501 8386 raft_consensus.cc:1261] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Refusing update from remote peer a9a7be24d9754b89b49b71e9783aa9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.242513 8246 raft_consensus.cc:1261] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Refusing update from remote peer a9a7be24d9754b89b49b71e9783aa9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:34.242724 8502 consensus_queue.cc:1035] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0896f644a79546128dcf15119d04e1c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.242817 8468 consensus_queue.cc:1035] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "086e23ca752a4a11975c79ed7076286e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:34.252985 7949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.253099 7949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:34.253716 7949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6c27d33cdbb4d6bad60b097d4a97e95: Sending DeleteTablet for 3 replicas of tablet fbaf128bf2364d89a51a2c5bea408d9a
I20260925 12:12:34.253970 8366 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59686
I20260925 12:12:34.254030 8085 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58486
I20260925 12:12:34.254060 8483 tablet_replica.cc:330] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: stopping tablet replica
I20260925 12:12:34.254113 8483 raft_consensus.cc:2229] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.254153 8483 raft_consensus.cc:2258] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.254150 8482 tablet_replica.cc:330] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca: stopping tablet replica
I20260925 12:12:34.254202 8482 raft_consensus.cc:2229] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:34.254225 8226 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf128bf2364d89a51a2c5bea408d9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42212
I20260925 12:12:34.254278 8482 raft_consensus.cc:2258] T fbaf128bf2364d89a51a2c5bea408d9a P a9a7be24d9754b89b49b71e9783aa9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.254300 8484 tablet_replica.cc:330] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: stopping tablet replica
I20260925 12:12:34.254341 8484 raft_consensus.cc:2229] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:34.254372 8484 raft_consensus.cc:2258] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:34.254688 8484 ts_tablet_manager.cc:1905] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.254730 8483 ts_tablet_manager.cc:1905] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:34.254808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8021
I20260925 12:12:34.256359 8484 ts_tablet_manager.cc:1918] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:12:34.256399 8484 log.cc:1196] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/fbaf128bf2364d89a51a2c5bea408d9a
I20260925 12:12:34.256425 8483 ts_tablet_manager.cc:1918] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 12:12:34.256477 8483 log.cc:1196] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/fbaf128bf2364d89a51a2c5bea408d9a
I20260925 12:12:34.256637 8484 ts_tablet_manager.cc:1939] T fbaf128bf2364d89a51a2c5bea408d9a P 086e23ca752a4a11975c79ed7076286e: Deleting consensus metadata
I20260925 12:12:34.256717 8483 ts_tablet_manager.cc:1939] T fbaf128bf2364d89a51a2c5bea408d9a P 0896f644a79546128dcf15119d04e1c5: Deleting consensus metadata
I20260925 12:12:34.256901 7936 catalog_manager.cc:4841] TS 086e23ca752a4a11975c79ed7076286e (127.0.98.2:37831): tablet fbaf128bf2364d89a51a2c5bea408d9a (table test-table [id=30f4596449544684a4015d547ba0af6d]) successfully deleted
I20260925 12:12:34.257112 7936 catalog_manager.cc:4841] TS 0896f644a79546128dcf15119d04e1c5 (127.0.98.3:42739): tablet fbaf128bf2364d89a51a2c5bea408d9a (table test-table [id=30f4596449544684a4015d547ba0af6d]) successfully deleted
W20260925 12:12:34.262799 7933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42411 (error 108)
I20260925 12:12:34.263019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8161
I20260925 12:12:34.263736 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:34.263177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.0.1:58524)
0925 12:12:34.263659 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:34.263702 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":47,"threads_started":1}
W20260925 12:12:34.263931 7933 catalog_manager.cc:4584] TS a9a7be24d9754b89b49b71e9783aa9ca (127.0.98.1:42411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbaf128bf2364d89a51a2c5bea408d9a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42411: connect: Connection refused (error 111)
I20260925 12:12:34.268765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8301
I20260925 12:12:34.274667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 7917
2026-09-25T12:12:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2498 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[8550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[8550](info): set up 2 sockets
Sep 25 12:12:34 dist-test-slave-2hqq krb5kdc[8550](info): commencing operation
krb5kdc: starting...
W20260925 12:12:35.308935 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:12:35 dist-test-slave-2hqq krb5kdc[8550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:12:35.428133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:35.509867 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:35.510108 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:35.510146 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:35.511379 8563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:35.511451 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:35.511477 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:35.511508 8563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:35.511533 8563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:35.513139 8563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:35.513427 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:35.513652 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:35.518463 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:35.518601 8563 server_base.cc:1034] running on GCE node
W20260925 12:12:35.518620 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:35.518479 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:35.519079 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:35.519392 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:35.520581 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790338355520543 us; error 53 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T12:12:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1250 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:12:35 dist-test-slave-2hqq krb5kdc[8576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:12:35 dist-test-slave-2hqq krb5kdc[8576](info): set up 2 sockets
Sep 25 12:12:35 dist-test-slave-2hqq krb5kdc[8576](info): commencing operation
krb5kdc: starting...
W20260925 12:12:36.564962 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:12:36 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:12:36.683578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:36.764163 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:36.764429 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:36.764465 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:36.765650 8589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:36.765723 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:36.765746 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:36.765769 8589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:36.765796 8589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:36.767262 8589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:36.767544 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:36.767805 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:36.772733 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:36.772744 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:36.772790 8589 server_base.cc:1034] running on GCE node
W20260925 12:12:36.772733 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:36.773130 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:36.773422 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:36.774586 8589 hybrid_clock.cc:648] HybridClock initialized: now 1790338356774564 us; error 40 us; skew 500 ppm
Sep 25 12:12:36 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:36.777124 8589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:12:36.777591 8589 webserver.cc:460] Webserver started at http://127.0.98.62:39289/ using document root <none> and password file <none>
I20260925 12:12:36.777822 8589 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:36.777865 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:36.778017 8589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:36.779129 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "e6ae8d8355d04ae4a01488e8dee294f2"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "446de535fe0e930cb5ac13986ea9d821"
server_key_iv: "f6b0f7bd9c30e6860eb79d2d31e80b89"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.779497 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6ae8d8355d04ae4a01488e8dee294f2"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "446de535fe0e930cb5ac13986ea9d821"
server_key_iv: "f6b0f7bd9c30e6860eb79d2d31e80b89"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.780877 8589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:36.781577 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:36.781764 8589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:36.781854 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e6ae8d8355d04ae4a01488e8dee294f2"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "446de535fe0e930cb5ac13986ea9d821"
server_key_iv: "f6b0f7bd9c30e6860eb79d2d31e80b89"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.781925 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:36.792917 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:36.794816 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:36.794953 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:36.799129 8589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37317
I20260925 12:12:36.799158 8656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37317 every 8 connection(s)
I20260925 12:12:36.799674 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:36.800307 8657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:36.802506 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Bootstrap starting.
I20260925 12:12:36.803138 8657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:36.803423 8657 log.cc:824] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:36.804116 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: No bootstrap required, opened a new log
I20260925 12:12:36.805727 8657 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:36.805866 8657 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:36.805887 8657 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ae8d8355d04ae4a01488e8dee294f2, State: Initialized, Role: FOLLOWER
I20260925 12:12:36.805986 8657 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:36.806077 8657 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:36.806120 8657 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:36.806167 8657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:36.807058 8657 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:36.807185 8657 leader_election.cc:302] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ae8d8355d04ae4a01488e8dee294f2; no voters:
I20260925 12:12:36.807363 8657 leader_election.cc:288] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:12:36.807416 8660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:36.807607 8660 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 LEADER]: Becoming Leader. State: Replica: e6ae8d8355d04ae4a01488e8dee294f2, State: Running, Role: LEADER
I20260925 12:12:36.807624 8657 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:36.807695 8660 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:36.807989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8589
I20260925 12:12:36.808127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:12:36.808377 392 external_mini_cluster.cc:1428] Setting key 6e47cf1fd424b9269f8639b24483f20b
I20260925 12:12:36.808395 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ae8d8355d04ae4a01488e8dee294f2. Latest consensus state: current_term: 1 leader_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260925 12:12:36.808519 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:36.808384 8661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260925 12:12:36.808674 8661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:36.808907 8670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:36.809478 8670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:36.810961 8670 catalog_manager.cc:1347] Generated new cluster ID: 9f31a9d996ad4a98ba58c2955d898cd0
I20260925 12:12:36.811009 8670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:12:36.815845 8670 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 12:12:36 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:36.816349 8670 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:36.820124 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:36.808979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.98.62:37317)
0925 12:12:36.809231 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:36.809241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:36.809307 (+ 66us) server_negotiation.cc:407] Connection header received
0925 12:12:36.809461 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:36.809467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:36.809546 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:36.809675 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:36.810417 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.811789 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.812856 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.813512 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.813626 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:36.816533 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:36.816537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:36.816538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:36.816550 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:36.818541 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:36.818931 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:36.818934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:36.818934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:36.819041 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:36.819244 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:36.819245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:36.819245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:36.819522 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:36.819683 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:36.819754 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:12:36.819898 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260925 12:12:36.820441 8670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Generated new TSK 0
I20260925 12:12:36.820608 8670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:12:36.851011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:36.931932 8683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:36.932135 8683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:36.932158 8683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:36.933338 8683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:36.933378 8683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:36.933403 8683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:36.934968 8683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:36.935245 8683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:36.935478 8683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:36.940174 8691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:36.940176 8688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:36.940201 8689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:36.940358 8683 server_base.cc:1034] running on GCE node
I20260925 12:12:36.940640 8683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:36.940873 8683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:36.942037 8683 hybrid_clock.cc:648] HybridClock initialized: now 1790338356942026 us; error 40 us; skew 500 ppm
Sep 25 12:12:36 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:36.944672 8683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:12:36.945052 8683 webserver.cc:460] Webserver started at http://127.0.98.1:42627/ using document root <none> and password file <none>
I20260925 12:12:36.945243 8683 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:36.945283 8683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:36.945416 8683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:36.946417 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d460d7e17f74ab39bbcda4d08cdde8d"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "e1c3230590625655dfbc1885d410a0e0"
server_key_iv: "8ebedd92e245924cd35ba5202d5ecdc6"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.946772 8683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d460d7e17f74ab39bbcda4d08cdde8d"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "e1c3230590625655dfbc1885d410a0e0"
server_key_iv: "8ebedd92e245924cd35ba5202d5ecdc6"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.948005 8683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:36.948761 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:36.948951 8683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:36.949036 8683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "0d460d7e17f74ab39bbcda4d08cdde8d"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "e1c3230590625655dfbc1885d410a0e0"
server_key_iv: "8ebedd92e245924cd35ba5202d5ecdc6"
server_key_version: "encryptionkey@0"
I20260925 12:12:36.949087 8683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:36.956094 8683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:36.958307 8683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:36.958419 8683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:36.958592 8683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:36.958910 8683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:36.958943 8683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:36.958966 8683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:36.958977 8683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:36.964752 8683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41701
I20260925 12:12:36.964800 8811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41701 every 8 connection(s)
I20260925 12:12:36.965101 8683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:36.965564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8683
I20260925 12:12:36.965703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:12:36.965920 392 external_mini_cluster.cc:1428] Setting key cbe9092fba487c7ff59632affe3a8aca
Sep 25 12:12:36 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:12:36.976258 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:36.966366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39187 (local address 127.0.98.62:37317)
0925 12:12:36.966432 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:36.966435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:36.966648 (+ 213us) server_negotiation.cc:407] Connection header received
0925 12:12:36.967966 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:36.967966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:36.967979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:36.968032 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:36.968823 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.969632 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.970723 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.971079 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.971172 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:36.973526 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:36.973528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:36.973528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:36.973536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:36.975021 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:36.975371 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:36.975372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:36.975372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:36.975418 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:36.975725 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:36.975726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:36.975726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:36.975831 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:36.975886 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:36.976127 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 12:12:36.976194 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:12:36.976428 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:36.966352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:39187)
0925 12:12:36.966523 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:36.966563 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:36.967862 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:36.968114 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:36.968119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:36.968467 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:36.968730 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:36.968735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.969725 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.969726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:36.970626 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:36.970628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:36.971188 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:36.971189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:36.971282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:36.971807 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:36.971815 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:36.973429 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:36.975090 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:36.975096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:36.975097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:36.975305 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:36.975474 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:36.975475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:36.975475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:36.975565 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:36.975890 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:36.975894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:36.976003 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 12:12:36.976084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:12:36.976204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260925 12:12:36.976905 8812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260925 12:12:36.976994 8812 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:36.977134 8812 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 12:12:36.977706 8621 ts_manager.cc:194] Registered new tserver with Master: 0d460d7e17f74ab39bbcda4d08cdde8d (127.0.98.1:41701)
I20260925 12:12:36.978629 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:12:36.991879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:37.072471 8817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.072690 8817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.072713 8817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.073788 8817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.073824 8817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.073849 8817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:37.075294 8817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.075474 8817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.075644 8817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.080490 8823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.080617 8817 server_base.cc:1034] running on GCE node
W20260925 12:12:37.080524 8822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.080529 8825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.080933 8817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.081238 8817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.082404 8817 hybrid_clock.cc:648] HybridClock initialized: now 1790338357082382 us; error 34 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.085022 8817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:12:37.085472 8817 webserver.cc:460] Webserver started at http://127.0.98.2:43293/ using document root <none> and password file <none>
I20260925 12:12:37.085733 8817 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.085803 8817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.085952 8817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:37.086990 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef293a70c9ba456ca7ef4941067f369c"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "67cd979cacc8552fa9e0eb0f34001ce4"
server_key_iv: "5b3e9b5228da8047c8a8789f08d8f332"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.087355 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef293a70c9ba456ca7ef4941067f369c"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "67cd979cacc8552fa9e0eb0f34001ce4"
server_key_iv: "5b3e9b5228da8047c8a8789f08d8f332"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.088732 8817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:37.089457 8832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.089628 8817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:37.089730 8817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "ef293a70c9ba456ca7ef4941067f369c"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "67cd979cacc8552fa9e0eb0f34001ce4"
server_key_iv: "5b3e9b5228da8047c8a8789f08d8f332"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.089808 8817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.102900 8817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.104758 8817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.104913 8817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.105131 8817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:37.105424 8817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:37.105479 8817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.105520 8817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:37.105547 8817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.111469 8817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37157
I20260925 12:12:37.111531 8945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37157 every 8 connection(s)
I20260925 12:12:37.111871 8817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:12:37.116459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8817
I20260925 12:12:37.116616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:37.116798 392 external_mini_cluster.cc:1428] Setting key 4de7bdb686e27f0583cac1251e2a36ce
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:37.121948 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.112597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41239 (local address 127.0.98.62:37317)
0925 12:12:37.112648 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.112650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.112913 (+ 263us) server_negotiation.cc:407] Connection header received
0925 12:12:37.113893 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.113895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.113907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.113964 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.114601 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.115105 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.116153 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.116365 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.116450 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.119035 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:37.119036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:37.119036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:37.119042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:37.120470 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.121002 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.121003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.121003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.121052 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.121319 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.121320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.121320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.121418 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:37.121474 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:37.121814 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 12:12:37.121885 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:12:37.122292 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.112620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:41239)
0925 12:12:37.112789 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.112824 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.113773 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.114032 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.114036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.114300 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.114493 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.114498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.115205 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.115206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.116054 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.116057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.116461 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.116462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.116644 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.117221 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.117234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.118934 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:37.120594 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:37.120600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:37.120602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:37.120926 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:37.121119 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:37.121120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:37.121120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:37.121257 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:37.121490 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:37.121496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:37.121665 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 12:12:37.121802 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 12:12:37.121987 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 12:12:37.122781 8946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260925 12:12:37.122886 8946 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:37.123078 8946 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
I20260925 12:12:37.123456 8621 ts_manager.cc:194] Registered new tserver with Master: ef293a70c9ba456ca7ef4941067f369c (127.0.98.2:37157)
I20260925 12:12:37.123840 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41239
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:12:37.143555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:37.223609 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.223822 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.223846 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.225049 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.225096 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.225154 8951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:37.226632 8951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.226874 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.227108 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.231854 8959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.231878 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.231892 8957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.232204 8951 server_base.cc:1034] running on GCE node
I20260925 12:12:37.232403 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.232671 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.233817 8951 hybrid_clock.cc:648] HybridClock initialized: now 1790338357233804 us; error 27 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.236255 8951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:12:37.236814 8951 webserver.cc:460] Webserver started at http://127.0.98.3:44641/ using document root <none> and password file <none>
I20260925 12:12:37.237044 8951 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.237118 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.237255 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:37.238294 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5f080b3219d45c8912666da13c4c6dd"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "e6de22e8781499fd4610e9e56d8b3ccf"
server_key_iv: "04ef2496682af0095635677e7caca7e1"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.238660 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5f080b3219d45c8912666da13c4c6dd"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "e6de22e8781499fd4610e9e56d8b3ccf"
server_key_iv: "04ef2496682af0095635677e7caca7e1"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.239969 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:37.240753 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.240929 8951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:37.241026 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "f5f080b3219d45c8912666da13c4c6dd"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "e6de22e8781499fd4610e9e56d8b3ccf"
server_key_iv: "04ef2496682af0095635677e7caca7e1"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.241129 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.258787 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.260896 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.261065 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.261319 8951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:37.261632 8951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:37.261687 8951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.261723 8951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:37.261770 8951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.267735 8951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33931
I20260925 12:12:37.267786 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33931 every 8 connection(s)
I20260925 12:12:37.268136 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:37.277467 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.268878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55719 (local address 127.0.98.62:37317)
0925 12:12:37.268926 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.268927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.269136 (+ 209us) server_negotiation.cc:407] Connection header received
0925 12:12:37.270075 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.270076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.270086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.270137 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.270816 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.271310 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.272283 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.272471 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.272591 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.274687 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:37.274688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:37.274689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:37.274696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:37.276144 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.276595 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.276596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.276596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.276659 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.276901 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.276901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.276902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.277001 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:37.277052 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:37.277339 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 12:12:37.277403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:12:37.277648 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.268860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:55719)
0925 12:12:37.269010 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.269047 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.269976 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.270195 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.270198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.270464 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.270660 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.270665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.271400 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.271401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.272196 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.272197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.272634 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.272635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.272735 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.273196 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.273205 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.274602 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:37.276213 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:37.276217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:37.276219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:37.276432 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:37.276741 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:37.276742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:37.276742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:37.276839 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:37.277076 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:37.277082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:37.277222 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 12:12:37.277311 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 12:12:37.277426 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":96,"threads_started":1}
I20260925 12:12:37.278126 9080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37317
I20260925 12:12:37.278205 9080 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:37.278270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 8951
I20260925 12:12:37.278376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:37.278415 9080 heartbeater.cc:511] Master 127.0.98.62:37317 requested a full tablet report, sending...
I20260925 12:12:37.278574 392 external_mini_cluster.cc:1428] Setting key ccf408c2523eb3d76c3ac3cf47a116e5
I20260925 12:12:37.278855 8621 ts_manager.cc:194] Registered new tserver with Master: f5f080b3219d45c8912666da13c4c6dd (127.0.98.3:33931)
I20260925 12:12:37.279277 8621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55719
I20260925 12:12:37.279462 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 12:12:37.347071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8683
I20260925 12:12:37.351255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8817
I20260925 12:12:37.355541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8951
I20260925 12:12:37.359397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 8589
I20260925 12:12:37.364279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--webserver_interface=127.0.98.62
--webserver_port=39289
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:37.442939 9089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.443140 9089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.443167 9089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.444344 9089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:37.444386 9089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.444401 9089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.444442 9089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:37.444473 9089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:37.446077 9089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.446324 9089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.446576 9089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.451340 9095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.451431 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.451368 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.451691 9089 server_base.cc:1034] running on GCE node
I20260925 12:12:37.451951 9089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.452263 9089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.453454 9089 hybrid_clock.cc:648] HybridClock initialized: now 1790338357453446 us; error 43 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.456555 9089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:12:37.457089 9089 webserver.cc:460] Webserver started at http://127.0.98.62:39289/ using document root <none> and password file <none>
I20260925 12:12:37.457350 9089 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.457427 9089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.458950 9089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:37.459550 9104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.459715 9089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.459781 9089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e6ae8d8355d04ae4a01488e8dee294f2"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "446de535fe0e930cb5ac13986ea9d821"
server_key_iv: "f6b0f7bd9c30e6860eb79d2d31e80b89"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.460093 9089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.482009 9089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.484294 9089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.484407 9089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.488201 9089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37317
I20260925 12:12:37.488284 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37317 every 8 connection(s)
I20260925 12:12:37.488804 9089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:37.490058 9157 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:12:37.490688 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Bootstrap starting.
I20260925 12:12:37.498637 9157 log.cc:824] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:37.498813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9089
I20260925 12:12:37.499080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41701
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37317
--webserver_port=42627
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:12:37.500541 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:12:37.500845 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Bootstrap complete.
I20260925 12:12:37.504645 9157 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:37.505025 9157 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ae8d8355d04ae4a01488e8dee294f2, State: Initialized, Role: FOLLOWER
I20260925 12:12:37.505224 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:37.505309 9157 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:37.505357 9157 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:37.505414 9157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:12:37.506986 9157 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:37.507123 9157 leader_election.cc:302] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6ae8d8355d04ae4a01488e8dee294f2; no voters:
I20260925 12:12:37.507310 9157 leader_election.cc:288] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:12:37.507385 9161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:12:37.507530 9161 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [term 2 LEADER]: Becoming Leader. State: Replica: e6ae8d8355d04ae4a01488e8dee294f2, State: Running, Role: LEADER
I20260925 12:12:37.507577 9157 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:37.507624 9161 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } }
I20260925 12:12:37.507962 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260925 12:12:37.508033 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:37.508260 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6ae8d8355d04ae4a01488e8dee294f2. Latest consensus state: current_term: 2 leader_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6ae8d8355d04ae4a01488e8dee294f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37317 } } }
I20260925 12:12:37.508390 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:37.508432 9167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:37.508940 9167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:37.509070 9167 catalog_manager.cc:1259] Loaded cluster ID: 9f31a9d996ad4a98ba58c2955d898cd0
I20260925 12:12:37.509292 9167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:12:37.510651 9167 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:37.510820 9167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e6ae8d8355d04ae4a01488e8dee294f2: Loaded TSK: 0
I20260925 12:12:37.511021 9167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:12:37.581581 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.581770 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.581799 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.582949 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.582991 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.583017 9160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:37.584394 9160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42627
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.584646 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.584887 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.589679 9186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.589722 9183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.589670 9184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.590562 9160 server_base.cc:1034] running on GCE node
I20260925 12:12:37.590811 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.591058 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.592217 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790338357592207 us; error 31 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.594717 9160 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 12:12:37.595196 9160 webserver.cc:460] Webserver started at http://127.0.98.1:42627/ using document root <none> and password file <none>
I20260925 12:12:37.595430 9160 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.595512 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.597115 9160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:37.597744 9193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.597913 9160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:37.597995 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "0d460d7e17f74ab39bbcda4d08cdde8d"
format_stamp: "Formatted at 2026-09-25 12:12:36 on dist-test-slave-2hqq"
server_key: "e1c3230590625655dfbc1885d410a0e0"
server_key_iv: "8ebedd92e245924cd35ba5202d5ecdc6"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.598348 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.613590 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.615871 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.616016 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.616285 9160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:37.616669 9160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:37.616704 9160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.616726 9160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:37.616771 9160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.624119 9160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41701
I20260925 12:12:37.624294 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41701 every 8 connection(s)
I20260925 12:12:37.624624 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.633208 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.625367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:51745)
0925 12:12:37.625543 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.625591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.626062 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.626483 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.626486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.627039 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.627218 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.627222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.628703 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.628704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.629475 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.629477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.630271 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.630272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.630363 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.630897 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.630905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.633069 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
W20260925 12:12:37.633339 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.625429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51745 (local address 127.0.98.62:37317)
0925 12:12:37.625641 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.625649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.625689 (+ 40us) server_negotiation.cc:407] Connection header received
0925 12:12:37.626245 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.626252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.626304 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.626438 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.627335 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.628603 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.629565 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.630184 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.630295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.633183 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51745: BlockingRecv error: recv got EOF from 127.0.98.1:51745 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
W20260925 12:12:37.633445 9307 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37317 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37317: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 12:12:37.634500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9160
I20260925 12:12:37.634800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37157
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37317
--webserver_port=43293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.637899 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.633656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:56687)
0925 12:12:37.633703 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.633714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.633750 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.633873 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.633873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.633974 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.634013 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.634014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.634638 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.634639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.635252 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.635254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.635622 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.635623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.635685 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.636115 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.636120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.637821 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:12:37.637961 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.633702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56687 (local address 127.0.98.62:37317)
0925 12:12:37.633751 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.633753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.633761 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:37.633805 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.633806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.633813 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.633861 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.634072 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.634529 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.635349 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.635546 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.635624 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.637889 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56687: BlockingRecv error: recv got EOF from 127.0.98.1:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.642799 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.638269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:49791)
0925 12:12:37.638383 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.638394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.638451 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.638586 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.638587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.638739 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.638774 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.638775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.639384 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.639385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.640262 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.640265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.640636 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.640637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.640701 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.641041 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.641047 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.642759 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 12:12:37.642966 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.638337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49791 (local address 127.0.98.62:37317)
0925 12:12:37.638386 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.638387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.638435 (+ 48us) server_negotiation.cc:407] Connection header received
0925 12:12:37.638510 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.638511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.638519 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.638568 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.638859 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.639307 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.640342 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.640572 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.640622 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.642885 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49791: BlockingRecv error: recv got EOF from 127.0.98.1:49791 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 12:12:37.643021 9307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:12:37.715782 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.715982 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.716005 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.717231 9311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.717271 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.717298 9311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:37.718760 9311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43293
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.718988 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.719266 9311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.724116 9316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.724102 9319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.724092 9317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.724290 9311 server_base.cc:1034] running on GCE node
I20260925 12:12:37.724455 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.724749 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.725911 9311 hybrid_clock.cc:648] HybridClock initialized: now 1790338357725907 us; error 36 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.728349 9311 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 12:12:37.728873 9311 webserver.cc:460] Webserver started at http://127.0.98.2:43293/ using document root <none> and password file <none>
I20260925 12:12:37.729105 9311 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.729188 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.730513 9311 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:37.731133 9326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.731283 9311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.731384 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "ef293a70c9ba456ca7ef4941067f369c"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "67cd979cacc8552fa9e0eb0f34001ce4"
server_key_iv: "5b3e9b5228da8047c8a8789f08d8f332"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.731729 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.754591 9311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.756603 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.756758 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.757002 9311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:37.757320 9311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:37.757354 9311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.757375 9311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:37.757422 9311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.763108 9311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37157
I20260925 12:12:37.763150 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37157 every 8 connection(s)
I20260925 12:12:37.763551 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.769984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9311
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.770294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33931
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37317
--webserver_port=44641
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:37.771131 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.764279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42605 (local address 127.0.98.62:37317)
0925 12:12:37.764352 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.764354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.764627 (+ 273us) server_negotiation.cc:407] Connection header received
0925 12:12:37.765164 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.765165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.765181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.765235 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.766165 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.766664 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.767662 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.767861 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.767956 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.771077 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42605: BlockingRecv error: recv got EOF from 127.0.98.2:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:12:37.771190 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.764279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:42605)
0925 12:12:37.764453 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.764529 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.765054 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.765303 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.765308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.765874 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.766065 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.766069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.766759 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.766760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.767571 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.767573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.767979 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.767981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.768104 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.768616 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.768624 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.770999 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260925 12:12:37.771670 9440 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37317 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37317: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.776770 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.771899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:32849)
0925 12:12:37.771985 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.771997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.772043 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.772184 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.772184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.772317 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.772363 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.772364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.773304 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.773305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.773884 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.773886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.774415 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.774416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.774471 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.774913 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.774918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.776724 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 12:12:37.776922 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.772031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32849 (local address 127.0.98.62:37317)
0925 12:12:37.772080 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.772082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.772089 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:37.772109 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.772110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.772119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.772173 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.772440 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.773221 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.773971 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.774319 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.774400 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.776838 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32849: BlockingRecv error: recv got EOF from 127.0.98.2:32849 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.781546 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.777027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:60213)
0925 12:12:37.777153 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.777165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.777203 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.777344 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.777345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.777446 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.777471 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.777472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.778393 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.778393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.778855 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.778856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.779352 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.779352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.779385 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.779789 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.779793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.781500 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260925 12:12:37.781661 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.777082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60213 (local address 127.0.98.62:37317)
0925 12:12:37.777185 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.777187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.777196 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:37.777259 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.777260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.777270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.777337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.777543 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.778317 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.778932 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.779271 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.779342 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.781600 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60213: BlockingRecv error: recv got EOF from 127.0.98.2:60213 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
W20260925 12:12:37.781713 9440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:12:37.854075 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:37.854266 9443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:37.854288 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:37.855391 9443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:37.855432 9443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:37.855458 9443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:37.856797 9443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44641
--tserver_master_addrs=127.0.98.62:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:37.856963 9443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:37.857125 9443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:37.862110 9453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:37.862118 9450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.862215 9443 server_base.cc:1034] running on GCE node
W20260925 12:12:37.862185 9451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:37.862479 9443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:37.862751 9443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:37.863901 9443 hybrid_clock.cc:648] HybridClock initialized: now 1790338357863892 us; error 31 us; skew 500 ppm
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338357, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:37.866361 9443 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 12:12:37.866799 9443 webserver.cc:460] Webserver started at http://127.0.98.3:44641/ using document root <none> and password file <none>
I20260925 12:12:37.867024 9443 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:37.867100 9443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:37.868551 9443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:37.869172 9460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.869390 9443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:37.869496 9443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "f5f080b3219d45c8912666da13c4c6dd"
format_stamp: "Formatted at 2026-09-25 12:12:37 on dist-test-slave-2hqq"
server_key: "e6de22e8781499fd4610e9e56d8b3ccf"
server_key_iv: "04ef2496682af0095635677e7caca7e1"
server_key_version: "encryptionkey@0"
I20260925 12:12:37.869866 9443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:37.886976 9443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:37.888953 9443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:37.889118 9443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:37.889353 9443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:37.889653 9443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:37.889706 9443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.889747 9443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:37.889775 9443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:37.895388 9443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33931
I20260925 12:12:37.895435 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33931 every 8 connection(s)
I20260925 12:12:37.895761 9443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:12:37.902673 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.896456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43783 (local address 127.0.98.62:37317)
0925 12:12:37.896530 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.896533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.896755 (+ 222us) server_negotiation.cc:407] Connection header received
0925 12:12:37.897210 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.897211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.897220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.897274 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.898170 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.898648 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.899610 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.899793 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.899842 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.902615 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43783: BlockingRecv error: recv got EOF from 127.0.98.3:43783 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 12:12:37.902679 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.896455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:43783)
0925 12:12:37.896621 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.896662 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.897127 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.897332 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.897335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.897874 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.898072 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.898076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.898725 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.898726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.899532 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.899534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.899860 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.899861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.899955 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.900368 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.900376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.902535 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
W20260925 12:12:37.902930 9574 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37317 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37317: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 12:12:37.905504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9443
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:37.907625 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.903140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:40731)
0925 12:12:37.903190 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.903201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.903246 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.903472 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.903473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.903595 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.903635 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.903636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.904218 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.904220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.904804 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.904806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.905176 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.905177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.905249 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.905714 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.905718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.907561 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:12:37.907732 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.903324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40731 (local address 127.0.98.62:37317)
0925 12:12:37.903371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.903372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.903378 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:12:37.903394 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.903395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.903401 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.903456 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.903708 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.904141 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.904897 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.905080 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.905177 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.907645 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40731: BlockingRecv error: recv got EOF from 127.0.98.3:40731 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:37 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 12:12:37.912771 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:37.908091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55949 (local address 127.0.98.62:37317)
0925 12:12:37.908132 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.908134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.908140 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:12:37.908160 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.908161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.908169 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.908231 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.908567 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.909149 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.910019 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.910337 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.910412 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.912723 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55949: BlockingRecv error: recv got EOF from 127.0.98.3:55949 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:12:37.912937 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.908020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:55949)
0925 12:12:37.908068 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:37.908078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:37.908114 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:37.908233 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:37.908233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:37.908433 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:37.908456 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.908458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.909214 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.909215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.909934 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:37.909936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.910415 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.910415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:37.910449 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.910834 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:37.910838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:37.912884 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:12:37.913046 9574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:12:37.913625 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:37.905918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.98.62:37317)
0925 12:12:37.905976 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:37.905978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:37.906148 (+ 170us) server_negotiation.cc:407] Connection header received
0925 12:12:37.906231 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:37.906232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:37.906242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:37.906314 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:37.907023 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.907768 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.908953 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:37.909170 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:37.909253 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:37.910522 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:37.910526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:37.910526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:37.910538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:37.912270 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.912670 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.912673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.912673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.912793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:37.913013 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:37.913014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:37.913015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:37.913286 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:37.913396 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:37.913485 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 12:12:37.913570 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:38.648252 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:38.643419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:59239)
0925 12:12:38.643582 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:38.643593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:38.643645 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:38.643830 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:38.643832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:38.644002 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:38.644053 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.644054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.644829 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.644830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.645389 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.645391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.645713 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.645714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.645752 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.646328 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:38.646333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:38.648186 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
W20260925 12:12:38.648348 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:38.643551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59239 (local address 127.0.98.62:37317)
0925 12:12:38.643693 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:38.643695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:38.643709 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:12:38.643730 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:38.643731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:38.643740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:38.643808 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:38.644121 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.644749 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.645460 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.645648 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.645700 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.648274 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59239: BlockingRecv error: recv got EOF from 127.0.98.1:59239 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:38.793839 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:38.782125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:59245)
0925 12:12:38.782416 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:38.782428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:38.782505 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:38.782658 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:38.782659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:38.782841 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:38.782888 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.782889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.783593 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.783594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.784204 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.784206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.784645 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.784646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.784709 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.791832 (+ 7123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:38.791838 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:38.793736 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
W20260925 12:12:38.793896 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:38.782210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59245 (local address 127.0.98.62:37317)
0925 12:12:38.782399 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:38.782401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:38.782501 (+ 100us) server_negotiation.cc:407] Connection header received
0925 12:12:38.782562 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:38.782563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:38.782575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:38.782652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:38.782983 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.783487 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.784299 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.784551 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.784643 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.793833 (+ 9190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59245: BlockingRecv error: recv got EOF from 127.0.98.2:59245 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:38 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:38.920331 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:38.913476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:54153)
0925 12:12:38.913613 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:38.913626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:38.913893 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:38.914083 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:38.914084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:38.914306 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:38.914367 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.914371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.915577 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.915578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.916555 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:38.916558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.917140 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.917141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:38.917210 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.917769 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:38.917775 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:38.920202 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260925 12:12:38.920563 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:38.913544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54153 (local address 127.0.98.62:37317)
0925 12:12:38.913613 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:38.913615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:38.913675 (+ 60us) server_negotiation.cc:407] Connection header received
0925 12:12:38.913990 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:38.913991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:38.914007 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:38.914075 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:38.914475 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.915479 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.916702 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:38.917030 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:38.917125 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:38.920315 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54153: BlockingRecv error: recv got EOF from 127.0.98.3:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:39.657200 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:39.648780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:38917)
0925 12:12:39.648916 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:39.648928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:39.648987 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:39.649151 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:39.649152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:39.649308 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:39.649363 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.649365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.650371 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.650372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.650903 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.650905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.651310 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.651311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.651368 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.655313 (+ 3945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:39.655317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:39.657093 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260925 12:12:39.657266 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:39.648898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38917 (local address 127.0.98.62:37317)
0925 12:12:39.649023 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:39.649025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:39.649033 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:39.649056 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:39.649056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:39.649065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:39.649134 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:39.649462 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.650286 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.651000 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.651237 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.651312 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.657174 (+ 5862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38917: BlockingRecv error: recv got EOF from 127.0.98.1:38917 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:39.798722 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:39.794340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:47197)
0925 12:12:39.794478 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:39.794488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:39.794554 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:39.794678 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:39.794679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:39.794832 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:39.794881 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.794883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.795509 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.795510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.796050 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.796051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.796378 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.796378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.796417 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.796959 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:39.796964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:39.798659 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
W20260925 12:12:39.798811 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:39.794403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47197 (local address 127.0.98.62:37317)
0925 12:12:39.794471 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:39.794473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:39.794557 (+ 84us) server_negotiation.cc:407] Connection header received
0925 12:12:39.794610 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:39.794611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:39.794621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:39.794662 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:39.794953 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.795433 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.796125 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.796301 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.796366 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.798747 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47197: BlockingRecv error: recv got EOF from 127.0.98.2:47197 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:39 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:39.925536 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:39.920819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:36339)
0925 12:12:39.920961 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:39.920972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:39.921035 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:39.921183 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:39.921184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:39.921378 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:39.921425 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.921427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.922116 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.922118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.922665 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:39.922667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.923034 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.923036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:39.923095 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.923533 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:39.923537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:39.925467 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
W20260925 12:12:39.925645 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:39.920961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36339 (local address 127.0.98.62:37317)
0925 12:12:39.921011 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:39.921013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:39.921023 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:39.921091 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:39.921092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:39.921105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:39.921177 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:39.921514 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.922019 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.922754 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:39.922945 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:39.923032 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:39.925553 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36339: BlockingRecv error: recv got EOF from 127.0.98.3:36339 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:40.662621 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:40.657658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:39377)
0925 12:12:40.657815 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:40.657826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:40.657897 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:40.658062 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:40.658063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:40.658273 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:40.658354 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.658356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.659000 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.659001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.659532 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.659534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.659866 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.659866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.659905 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.660395 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:40.660400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:40.662563 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
W20260925 12:12:40.662803 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:40.657705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39377 (local address 127.0.98.62:37317)
0925 12:12:40.657825 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:40.657827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:40.657898 (+ 71us) server_negotiation.cc:407] Connection header received
0925 12:12:40.657952 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:40.657952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:40.657966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:40.658044 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:40.658437 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.658933 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.659606 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.659801 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.659853 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.662700 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39377: BlockingRecv error: recv got EOF from 127.0.98.1:39377 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:40.803786 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:40.799282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:32779)
0925 12:12:40.799416 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:40.799427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:40.799496 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:40.799634 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:40.799634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:40.799796 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:40.799845 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.799847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.800482 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.800483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.801137 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.801140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.801470 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.801471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.801509 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.801956 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:40.801962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:40.803728 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260925 12:12:40.803897 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:40.799347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32779 (local address 127.0.98.62:37317)
0925 12:12:40.799395 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:40.799397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:40.799494 (+ 97us) server_negotiation.cc:407] Connection header received
0925 12:12:40.799551 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:40.799552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:40.799563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:40.799613 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:40.799910 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.800406 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.801205 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.801404 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.801460 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.803823 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32779: BlockingRecv error: recv got EOF from 127.0.98.2:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:40 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:40.930593 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:40.926015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:57827)
0925 12:12:40.926186 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:40.926204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:40.926281 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:40.926427 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:40.926428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:40.926604 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:40.926649 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.926651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.927303 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.927305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.927849 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:40.927851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.928247 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.928248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:40.928312 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.928774 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:40.928779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:40.930495 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
W20260925 12:12:40.930634 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:40.926081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57827 (local address 127.0.98.62:37317)
0925 12:12:40.926143 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:40.926146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:40.926260 (+ 114us) server_negotiation.cc:407] Connection header received
0925 12:12:40.926343 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:40.926344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:40.926357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:40.926409 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:40.926727 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.927224 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.927934 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:40.928156 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:40.928245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:40.930572 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57827: BlockingRecv error: recv got EOF from 127.0.98.3:57827 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:41.667441 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:41.663020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:39363)
0925 12:12:41.663202 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:41.663216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:41.663262 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:41.663429 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:41.663430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:41.663581 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:41.663631 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.663633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.664245 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.664246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.664827 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.664830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.665185 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.665186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.665234 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.665663 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:41.665668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:41.667347 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
W20260925 12:12:41.667492 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:41.663089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39363 (local address 127.0.98.62:37317)
0925 12:12:41.663237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:41.663239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:41.663262 (+ 23us) server_negotiation.cc:407] Connection header received
0925 12:12:41.663364 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:41.663365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:41.663375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:41.663426 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:41.663693 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.664179 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.664898 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.665090 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.665153 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.667421 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39363: BlockingRecv error: recv got EOF from 127.0.98.1:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:41.809343 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:41.804361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:58885)
0925 12:12:41.804542 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:41.804578 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:41.804651 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:41.804820 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:41.804821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:41.805001 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:41.805059 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.805061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.805740 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.805742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.806271 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.806272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.806650 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.806651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.806712 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.807286 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:41.807291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:41.809246 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
W20260925 12:12:41.809391 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:41.804530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58885 (local address 127.0.98.62:37317)
0925 12:12:41.804597 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:41.804599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:41.804631 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:12:41.804719 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:41.804719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:41.804738 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:41.804797 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:41.805155 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.805654 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.806368 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.806567 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.806649 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.809324 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58885: BlockingRecv error: recv got EOF from 127.0.98.2:58885 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:41 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:41.935478 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:41.931033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:35345)
0925 12:12:41.931197 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:41.931207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:41.931260 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:41.931381 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:41.931381 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:41.931533 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:41.931580 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.931581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.932189 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.932190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.932760 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:41.932762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.933063 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.933064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:41.933108 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.933602 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:41.933607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:41.935421 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
W20260925 12:12:41.935631 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:41.931123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35345 (local address 127.0.98.62:37317)
0925 12:12:41.931185 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:41.931187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:41.931263 (+ 76us) server_negotiation.cc:407] Connection header received
0925 12:12:41.931299 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:41.931299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:41.931310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:41.931360 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:41.931645 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.932123 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.932829 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:41.933003 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:41.933052 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:41.935536 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35345: BlockingRecv error: recv got EOF from 127.0.98.3:35345 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:42.672920 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:42.667899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:57541)
0925 12:12:42.668036 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:42.668046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:42.668099 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:42.668273 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:42.668273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:42.668428 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:42.668548 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.668550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.669182 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.669183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.669739 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.669741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.670066 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.670067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.670111 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.670553 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:42.670558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:42.672813 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260925 12:12:42.672966 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:42.668007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57541 (local address 127.0.98.62:37317)
0925 12:12:42.668145 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:42.668147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:42.668156 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:42.668186 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:42.668186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:42.668196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:42.668254 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:42.668612 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.669110 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.669810 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.669999 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.670050 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.672891 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57541: BlockingRecv error: recv got EOF from 127.0.98.1:57541 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:42.814514 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:42.809822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:41885)
0925 12:12:42.810022 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:42.810035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:42.810104 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:42.810241 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:42.810242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:42.810407 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:42.810456 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.810457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.811159 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.811160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.811695 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.811697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.812077 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.812079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.812122 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.812605 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:42.812610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:42.814421 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
W20260925 12:12:42.814626 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:42.809997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41885 (local address 127.0.98.62:37317)
0925 12:12:42.810057 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:42.810059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:42.810089 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:12:42.810170 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:42.810171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:42.810181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:42.810221 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:42.810556 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.811063 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.811777 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.811971 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.812051 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.814538 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41885: BlockingRecv error: recv got EOF from 127.0.98.2:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:42 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:42.940644 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:42.935887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:35519)
0925 12:12:42.936019 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:42.936029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:42.936082 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:42.936223 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:42.936224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:42.936380 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:42.936429 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.936430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.937152 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.937153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.937793 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:42.937796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.938190 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.938191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:42.938248 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.938700 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:42.938705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:42.940545 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
W20260925 12:12:42.940660 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:42.935973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35519 (local address 127.0.98.62:37317)
0925 12:12:42.936047 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:42.936048 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:42.936068 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:12:42.936139 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:42.936139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:42.936151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:42.936205 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:42.936536 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.937079 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.937888 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:42.938087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:42.938189 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:42.940602 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35519: BlockingRecv error: recv got EOF from 127.0.98.3:35519 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:43.678052 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:43.673458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:45237)
0925 12:12:43.673607 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:43.673620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:43.673692 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:43.673864 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:43.673865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:43.674066 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:43.674114 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.674115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.674835 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.674836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.675379 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.675380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.675728 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.675729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.675770 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.676216 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:43.676220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:43.677994 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260925 12:12:43.678194 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:43.673536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45237 (local address 127.0.98.62:37317)
0925 12:12:43.673652 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:43.673654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:43.673674 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:12:43.673745 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:43.673745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:43.673755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:43.673803 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:43.674214 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.674758 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.675464 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.675658 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.675717 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.678099 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45237: BlockingRecv error: recv got EOF from 127.0.98.1:45237 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:43.819756 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:43.815161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:51293)
0925 12:12:43.815292 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:43.815306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:43.815383 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:43.815543 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:43.815544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:43.815784 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:43.815848 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.815851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.816563 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.816566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.817116 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.817118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.817453 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.817454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.817492 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.817942 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:43.817947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:43.819695 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
W20260925 12:12:43.819954 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:43.815228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51293 (local address 127.0.98.62:37317)
0925 12:12:43.815298 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:43.815300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:43.815410 (+ 110us) server_negotiation.cc:407] Connection header received
0925 12:12:43.815444 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:43.815445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:43.815458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:43.815518 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:43.815970 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.816465 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.817186 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.817388 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.817441 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.819868 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51293: BlockingRecv error: recv got EOF from 127.0.98.2:51293 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:43 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:43.946357 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:43.941192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:56125)
0925 12:12:43.941421 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:43.941434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:43.941566 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:43.941716 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:43.941717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:43.941893 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:43.941940 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.941942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.942678 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.942679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.943231 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:43.943233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.943595 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.943596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:43.943634 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.944110 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:43.944114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:43.946257 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":169,"threads_started":1}
W20260925 12:12:43.946420 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:43.941320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56125 (local address 127.0.98.62:37317)
0925 12:12:43.941385 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:43.941387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:43.941562 (+ 175us) server_negotiation.cc:407] Connection header received
0925 12:12:43.941634 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:43.941635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:43.941647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:43.941694 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:43.942016 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.942571 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.943319 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:43.943525 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:43.943586 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:43.946351 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56125: BlockingRecv error: recv got EOF from 127.0.98.3:56125 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:44.687114 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:44.678557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:52647)
0925 12:12:44.678741 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:44.678751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:44.678809 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:44.678953 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:44.678954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:44.679110 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:44.679158 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.679159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.679858 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.679858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.680368 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.680369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.680727 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.680728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.680772 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.685066 (+ 4294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:44.685071 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:44.687049 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
W20260925 12:12:44.687259 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:44.678639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52647 (local address 127.0.98.62:37317)
0925 12:12:44.678748 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:44.678752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:44.678820 (+ 68us) server_negotiation.cc:407] Connection header received
0925 12:12:44.678863 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:44.678864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:44.678875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:44.678940 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:44.679220 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.679790 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.680440 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.680664 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.680719 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.687166 (+ 6447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52647: BlockingRecv error: recv got EOF from 127.0.98.1:52647 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:44.825179 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:44.820259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:54121)
0925 12:12:44.820446 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:44.820463 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:44.820552 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:44.820695 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:44.820696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:44.820883 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:44.820934 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.820936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.821570 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.821571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.822160 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.822162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.822488 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.822488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.822529 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.823077 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:44.823091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:44.825085 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
W20260925 12:12:44.825227 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:44.820333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54121 (local address 127.0.98.62:37317)
0925 12:12:44.820447 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:44.820449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:44.820549 (+ 100us) server_negotiation.cc:407] Connection header received
0925 12:12:44.820602 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:44.820602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:44.820612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:44.820672 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:44.820997 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.821496 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.822234 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.822421 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.822470 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.825167 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54121: BlockingRecv error: recv got EOF from 127.0.98.2:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:44 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:44.952198 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:44.946966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:54201)
0925 12:12:44.947143 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:44.947158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:44.947262 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:44.947433 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:44.947434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:44.947626 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:44.947698 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.947700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.948772 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.948773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.949329 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:44.949331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.949702 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.949703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:44.949743 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.950212 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:44.950216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:44.952132 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
W20260925 12:12:44.952332 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:44.947044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54201 (local address 127.0.98.62:37317)
0925 12:12:44.947118 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:44.947120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:44.947230 (+ 110us) server_negotiation.cc:407] Connection header received
0925 12:12:44.947348 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:44.947349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:44.947365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:44.947417 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:44.947880 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.948697 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.949435 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:44.949626 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:44.949696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:44.952224 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54201: BlockingRecv error: recv got EOF from 127.0.98.3:54201 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:45.692230 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:45.687652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:44827)
0925 12:12:45.687835 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:45.687846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:45.687900 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:45.688064 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:45.688065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:45.688229 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:45.688277 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.688280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.689055 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.689056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.689588 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.689589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.689945 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.689946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.689985 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.690494 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:45.690499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:45.692165 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":111,"threads_started":1}
W20260925 12:12:45.692337 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:45.687791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44827 (local address 127.0.98.62:37317)
0925 12:12:45.687933 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:45.687935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:45.687944 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:45.687972 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:45.687972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:45.687983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:45.688048 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:45.688370 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.688980 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.689694 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.689880 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.689932 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.692264 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44827: BlockingRecv error: recv got EOF from 127.0.98.1:44827 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:45.831033 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:45.825753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:32801)
0925 12:12:45.825934 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:45.825955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:45.826026 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:45.826144 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:45.826145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:45.826323 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:45.826371 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.826373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.827410 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.827411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.828055 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.828058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.828383 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.828384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.828427 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.829052 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:45.829056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:45.830971 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
W20260925 12:12:45.831193 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:45.825889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32801 (local address 127.0.98.62:37317)
0925 12:12:45.825960 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:45.825962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:45.826007 (+ 45us) server_negotiation.cc:407] Connection header received
0925 12:12:45.826078 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:45.826078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:45.826088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:45.826136 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:45.826469 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.827338 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.828136 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.828318 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.828367 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.831114 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32801: BlockingRecv error: recv got EOF from 127.0.98.2:32801 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:45 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:45.958367 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:45.952721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:50395)
0925 12:12:45.952862 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:45.952877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:45.952939 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:45.953080 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:45.953081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:45.953257 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:45.953302 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.953303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.954044 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.954045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.955082 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:45.955085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.955627 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.955628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:45.955707 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.956278 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:45.956283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:45.958284 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260925 12:12:45.958477 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:45.952772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50395 (local address 127.0.98.62:37317)
0925 12:12:45.952845 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:45.952847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:45.952937 (+ 90us) server_negotiation.cc:407] Connection header received
0925 12:12:45.952990 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:45.952990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:45.953006 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:45.953059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:45.953369 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.953951 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.955198 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:45.955546 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:45.955618 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:45.958406 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50395: BlockingRecv error: recv got EOF from 127.0.98.3:50395 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:46.697628 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:46.692783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:33615)
0925 12:12:46.692919 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:46.692931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:46.692985 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:46.693159 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:46.693159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:46.693333 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:46.693382 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.693384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.694119 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.694120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.694644 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.694645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.694996 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.694997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.695039 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.695557 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:46.695564 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:46.697534 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260925 12:12:46.697705 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:46.692899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33615 (local address 127.0.98.62:37317)
0925 12:12:46.693036 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:46.693038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:46.693045 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:46.693066 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:46.693066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:46.693076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:46.693139 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:46.693485 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.694034 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.694724 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.694925 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.694983 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.697630 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33615: BlockingRecv error: recv got EOF from 127.0.98.1:33615 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:46.836686 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:46.831657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:43665)
0925 12:12:46.831815 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:46.831827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:46.831886 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:46.832076 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:46.832077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:46.832243 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:46.832294 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.832295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.833093 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.833094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.833664 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.833666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.833984 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.833985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.834076 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.834600 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:46.834605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:46.836621 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
W20260925 12:12:46.836829 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:46.831872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43665 (local address 127.0.98.62:37317)
0925 12:12:46.831949 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:46.831951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:46.831959 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:46.831981 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:46.831981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:46.831992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:46.832051 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:46.832402 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.833006 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.833742 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.833921 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.833973 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.836736 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43665: BlockingRecv error: recv got EOF from 127.0.98.2:43665 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:46 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:46.963887 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:46.958969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:36015)
0925 12:12:46.959125 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:46.959136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:46.959193 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:46.959335 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:46.959335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:46.959497 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:46.959546 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.959548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.960205 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.960206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.960745 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:46.960747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.961170 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.961171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:46.961241 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.961852 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:46.961857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:46.963771 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260925 12:12:46.963922 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:46.959061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36015 (local address 127.0.98.62:37317)
0925 12:12:46.959139 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:46.959140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:46.959195 (+ 55us) server_negotiation.cc:407] Connection header received
0925 12:12:46.959239 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:46.959240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:46.959260 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:46.959315 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:46.959616 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.960129 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.960822 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:46.961049 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:46.961142 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:46.963860 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36015: BlockingRecv error: recv got EOF from 127.0.98.3:36015 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:47.703305 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:47.698156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:42579)
0925 12:12:47.698312 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:47.698324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:47.698376 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:47.698529 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:47.698530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:47.698690 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:47.698736 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.698737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.699753 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.699754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.700403 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.700405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.700853 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.700854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.700896 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.701339 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:47.701343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:47.703246 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260925 12:12:47.703428 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:47.698243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42579 (local address 127.0.98.62:37317)
0925 12:12:47.698352 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:47.698354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:47.698364 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:47.698438 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:47.698439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:47.698449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:47.698514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:47.698813 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.699588 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.700568 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.700785 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.700845 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.703343 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42579: BlockingRecv error: recv got EOF from 127.0.98.1:42579 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:47.841885 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:47.837209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:39111)
0925 12:12:47.837398 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:47.837408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:47.837464 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:47.837597 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:47.837598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:47.837761 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:47.837810 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.837811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.838445 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.838445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.839243 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.839246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.839582 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.839583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.839623 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.840107 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:47.840113 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:47.841828 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
W20260925 12:12:47.841988 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:47.837300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39111 (local address 127.0.98.62:37317)
0925 12:12:47.837364 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:47.837366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:47.837471 (+ 105us) server_negotiation.cc:407] Connection header received
0925 12:12:47.837513 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:47.837514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:47.837526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:47.837578 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:47.837871 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.838364 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.839326 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.839518 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.839570 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.841918 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39111: BlockingRecv error: recv got EOF from 127.0.98.2:39111 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:47 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:47.969178 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:47.964388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:33065)
0925 12:12:47.964647 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:47.964657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:47.964712 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:47.964828 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:47.964829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:47.964989 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:47.965038 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.965039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.965692 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.965693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.966224 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:47.966226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.966651 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.966652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:47.966703 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.967126 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:47.967131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:47.969121 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":149,"threads_started":1}
W20260925 12:12:47.969328 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:47.964562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33065 (local address 127.0.98.62:37317)
0925 12:12:47.964617 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:47.964619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:47.964701 (+ 82us) server_negotiation.cc:407] Connection header received
0925 12:12:47.964757 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:47.964758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:47.964770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:47.964825 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:47.965095 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.965611 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.966310 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:47.966556 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:47.966623 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:47.969245 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33065: BlockingRecv error: recv got EOF from 127.0.98.3:33065 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:48.709178 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:48.703791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:36859)
0925 12:12:48.703945 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:48.703957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:48.704022 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:48.704214 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:48.704215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:48.704390 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:48.704439 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.704441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.705182 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.705183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.705937 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.705940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.706443 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.706444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.706511 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.707083 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:48.707089 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:48.709119 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260925 12:12:48.709417 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:48.703871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36859 (local address 127.0.98.62:37317)
0925 12:12:48.704002 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:48.704004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:48.704013 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:12:48.704065 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:48.704066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:48.704076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:48.704140 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:48.704564 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.705085 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.706055 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.706345 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.706441 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.709236 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36859: BlockingRecv error: recv got EOF from 127.0.98.1:36859 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:48.847329 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:48.842407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:56741)
0925 12:12:48.842545 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:48.842559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:48.842633 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:48.842784 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:48.842786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:48.842965 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:48.843018 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.843020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.843857 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.843859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.844527 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.844530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.844958 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.844959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.845008 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.845438 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:48.845442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:48.847247 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260925 12:12:48.847443 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:48.842499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56741 (local address 127.0.98.62:37317)
0925 12:12:48.842561 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:48.842563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:48.842623 (+ 60us) server_negotiation.cc:407] Connection header received
0925 12:12:48.842693 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:48.842693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:48.842703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:48.842763 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:48.843161 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.843754 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.844640 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.844871 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.844928 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.847353 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56741: BlockingRecv error: recv got EOF from 127.0.98.2:56741 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:48 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:48.976014 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:48.969619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:43967)
0925 12:12:48.969776 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:48.969788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:48.969850 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:48.970014 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:48.970015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:48.970203 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:48.970250 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.970252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.971003 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.971004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.971906 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:48.971910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.972562 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.972563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:48.972641 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.973262 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:48.973286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:48.975911 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
W20260925 12:12:48.976058 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:48.969691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43967 (local address 127.0.98.62:37317)
0925 12:12:48.969760 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:48.969763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:48.969840 (+ 77us) server_negotiation.cc:407] Connection header received
0925 12:12:48.969926 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:48.969928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:48.969941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:48.970006 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:48.970342 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.970899 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.971985 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:48.972422 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:48.972543 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:48.975989 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43967: BlockingRecv error: recv got EOF from 127.0.98.3:43967 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:49.721869 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:49.709643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:59521)
0925 12:12:49.709795 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:49.709823 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:49.709875 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:49.710004 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:49.710005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:49.710160 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:49.710208 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.710209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.710891 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.710892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.711408 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.711409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.711729 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.711729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.711767 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.719784 (+ 8017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:49.719791 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:49.721809 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260925 12:12:49.721992 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:49.709744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59521 (local address 127.0.98.62:37317)
0925 12:12:49.709885 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:49.709887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:49.709895 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:49.709918 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:49.709919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:49.709929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:49.709991 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:49.710315 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.710823 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.711480 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.711668 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.711719 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.721908 (+ 10189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59521: BlockingRecv error: recv got EOF from 127.0.98.1:59521 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:49.852890 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:49.847842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:56749)
0925 12:12:49.847959 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:49.847976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:49.848030 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:49.848200 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:49.848201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:49.848424 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:49.848535 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.848539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.849335 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.849336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.849870 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.849871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.850201 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.850202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.850247 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.850723 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:49.850729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:49.852820 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
W20260925 12:12:49.853016 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:49.847908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56749 (local address 127.0.98.62:37317)
0925 12:12:49.848008 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:49.848010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:49.848020 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:49.848097 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:49.848098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:49.848108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:49.848164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:49.848620 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.849250 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.849948 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.850130 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.850183 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.852934 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56749: BlockingRecv error: recv got EOF from 127.0.98.2:56749 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:49 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:49.981885 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:49.976969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:42431)
0925 12:12:49.977156 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:49.977168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:49.977232 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:49.977366 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:49.977367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:49.977529 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:49.977577 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.977580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.978235 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.978236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.978767 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:49.978768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.979098 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.979098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:49.979137 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.979662 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:49.979667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:49.981777 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
W20260925 12:12:49.981930 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:49.977053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42431 (local address 127.0.98.62:37317)
0925 12:12:49.977151 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:49.977153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:49.977229 (+ 76us) server_negotiation.cc:407] Connection header received
0925 12:12:49.977281 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:49.977282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:49.977294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:49.977344 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:49.977638 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.978158 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.978844 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:49.979032 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:49.979087 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:49.981853 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42431: BlockingRecv error: recv got EOF from 127.0.98.3:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:50.727084 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:50.722377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:57401)
0925 12:12:50.722533 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:50.722543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:50.722593 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:50.722730 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:50.722731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:50.722890 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:50.722939 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.722940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.723592 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.723593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.724108 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.724109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.724678 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.724679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.724753 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.725274 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:50.725279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:50.727012 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260925 12:12:50.727286 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:50.722462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57401 (local address 127.0.98.62:37317)
0925 12:12:50.722597 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:50.722598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:50.722609 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:12:50.722632 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:50.722633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:50.722643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:50.722714 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:50.723007 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.723518 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.724214 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.724564 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.724635 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.727197 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57401: BlockingRecv error: recv got EOF from 127.0.98.1:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:50.858273 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:50.853449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:52357)
0925 12:12:50.853595 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:50.853608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:50.853681 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:50.853858 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:50.853859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:50.854040 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:50.854085 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.854136 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.854945 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.854947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.855512 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.855514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.855902 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.855903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.855962 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.856416 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:50.856421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:50.858176 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260925 12:12:50.858307 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:50.853590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52357 (local address 127.0.98.62:37317)
0925 12:12:50.853647 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:50.853649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:50.853659 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:50.853748 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:50.853749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:50.853768 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:50.853826 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:50.854224 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.854872 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.855605 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.855812 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.855900 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.858250 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52357: BlockingRecv error: recv got EOF from 127.0.98.2:52357 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:50 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:50.987210 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:50.982519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:34833)
0925 12:12:50.982658 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:50.982677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:50.982736 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:50.982893 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:50.982894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:50.983061 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:50.983110 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.983111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.983767 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.983768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.984420 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:50.984423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.984818 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.984819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:50.984861 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.985320 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:50.985324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:50.987055 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260925 12:12:50.987298 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:50.982591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34833 (local address 127.0.98.62:37317)
0925 12:12:50.982637 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:50.982639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:50.982756 (+ 117us) server_negotiation.cc:407] Connection header received
0925 12:12:50.982808 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:50.982808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:50.982821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:50.982873 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:50.983178 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.983675 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.984505 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:50.984752 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:50.984805 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:50.987213 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34833: BlockingRecv error: recv got EOF from 127.0.98.3:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:51.732803 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:51.727638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:59779)
0925 12:12:51.727810 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:51.727827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:51.727875 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:51.728137 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:51.728138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:51.728298 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:51.728348 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.728350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.729091 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.729092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.729605 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.729606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.729934 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.729935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.729975 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.730429 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:51.730433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:51.732712 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
W20260925 12:12:51.732880 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:51.727842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59779 (local address 127.0.98.62:37317)
0925 12:12:51.727982 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:51.727984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:51.727992 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:51.728015 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:51.728016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:51.728026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:51.728092 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:51.728420 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.728991 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.729683 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.729870 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.729921 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.732826 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59779: BlockingRecv error: recv got EOF from 127.0.98.1:59779 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:51.863727 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:51.858814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:42689)
0925 12:12:51.858960 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:51.858978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:51.859036 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:51.859182 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:51.859183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:51.859343 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:51.859392 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.859393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.860124 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.860124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.860707 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.860709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.861080 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.861081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.861124 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.861688 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:51.861696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:51.863657 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260925 12:12:51.863884 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:51.858907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42689 (local address 127.0.98.62:37317)
0925 12:12:51.858979 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:51.858981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:51.859023 (+ 42us) server_negotiation.cc:407] Connection header received
0925 12:12:51.859112 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:51.859113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:51.859123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:51.859176 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:51.859497 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.860004 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.860794 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.861007 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.861067 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.863778 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42689: BlockingRecv error: recv got EOF from 127.0.98.2:42689 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:51 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:51.992447 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:51.987842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.3:39943)
0925 12:12:51.987997 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:51.988014 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:51.988074 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:51.988207 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:51.988207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:51.988370 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:51.988416 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.988418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.989108 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.989108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.989651 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:51.989653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.989959 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.989960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:51.989999 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.990499 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:51.990504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:51.992380 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
W20260925 12:12:51.992604 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:51.987913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39943 (local address 127.0.98.62:37317)
0925 12:12:51.987979 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:51.987981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:51.988080 (+ 99us) server_negotiation.cc:407] Connection header received
0925 12:12:51.988130 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:51.988131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:51.988144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:51.988186 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:51.988480 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.989039 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.989723 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:51.989897 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:51.989946 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:51.992463 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39943: BlockingRecv error: recv got EOF from 127.0.98.3:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 12:12:52 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:52 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:52.738262 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:52.733357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.1:43697)
0925 12:12:52.733513 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:52.733524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:52.733570 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:52.733696 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:52.733697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:52.733929 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:52.733978 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:52.733980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.734677 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.734678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:52.735377 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:52.735379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.735763 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.735764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:52.735813 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:52.736337 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:52.736342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:52.738194 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
W20260925 12:12:52.738416 9635 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:52.733452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43697 (local address 127.0.98.62:37317)
0925 12:12:52.733581 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:52.733583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:52.733590 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:52.733611 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:52.733611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:52.733621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:52.733681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:52.734040 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.734574 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.735479 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.735681 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.735735 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:52.738310 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43697: BlockingRecv error: recv got EOF from 127.0.98.1:43697 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
Sep 25 12:12:52 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 12:12:52 dist-test-slave-2hqq krb5kdc[8576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 12:12:52.868769 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:52.864293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37317 (local address 127.0.98.2:39179)
0925 12:12:52.864438 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:52.864451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:52.864549 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:52.864687 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:52.864688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:52.864837 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:52.864885 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:52.864886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.865546 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.865547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:52.866110 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:52.866112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.866423 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.866424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:52.866465 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:52.866899 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:52.866903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:52.868703 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37317: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260925 12:12:52.868839 9635 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:52.864361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39179 (local address 127.0.98.62:37317)
0925 12:12:52.864427 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:52.864429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:52.864520 (+ 91us) server_negotiation.cc:407] Connection header received
0925 12:12:52.864604 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:52.864605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:52.864615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:52.864670 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:52.864948 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.865475 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.866181 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:52.866362 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:52.866411 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:52.868795 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39179: BlockingRecv error: recv got EOF from 127.0.98.2:39179 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:12:52.915930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9160
I20260925 12:12:52.920284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9311
I20260925 12:12:52.924410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9443
I20260925 12:12:52.928882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9089
2026-09-25T12:12:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17402 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T12:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:52Z Disabled control of system clock
I20260925 12:12:52.956759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39597
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 12:12:53.038569 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.038775 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.038795 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.039897 9640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:53.039932 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.039944 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.039958 9640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:53.039969 9640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:53.041247 9640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.041445 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.041685 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.047102 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.047185 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.047168 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.047153 9640 server_base.cc:1034] running on GCE node
I20260925 12:12:53.047683 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.048014 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.049183 9640 hybrid_clock.cc:648] HybridClock initialized: now 1790338373049168 us; error 28 us; skew 500 ppm
I20260925 12:12:53.050518 9640 webserver.cc:460] Webserver started at http://127.0.98.62:45057/ using document root <none> and password file <none>
I20260925 12:12:53.050767 9640 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.050834 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.051008 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.052110 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "355c47f6f91a49b399eb6c9ed07fda78"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "e93885a9e96f762795bacaca00e94afb"
server_key_iv: "9191de6b825b65f4eff41aa22504a3ef"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.052477 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "355c47f6f91a49b399eb6c9ed07fda78"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "e93885a9e96f762795bacaca00e94afb"
server_key_iv: "9191de6b825b65f4eff41aa22504a3ef"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.053855 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:53.054646 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.054821 9640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:53.054925 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "355c47f6f91a49b399eb6c9ed07fda78"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "e93885a9e96f762795bacaca00e94afb"
server_key_iv: "9191de6b825b65f4eff41aa22504a3ef"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.055047 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.088748 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.089110 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.089270 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.093336 9640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39597
I20260925 12:12:53.093384 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39597 every 8 connection(s)
I20260925 12:12:53.093743 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:53.094398 9707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:53.097121 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78: Bootstrap starting.
I20260925 12:12:53.097736 9707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:53.098045 9707 log.cc:824] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:53.098717 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78: No bootstrap required, opened a new log
I20260925 12:12:53.100291 9707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } }
I20260925 12:12:53.100428 9707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:53.100450 9707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355c47f6f91a49b399eb6c9ed07fda78, State: Initialized, Role: FOLLOWER
I20260925 12:12:53.100683 9707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } }
I20260925 12:12:53.100772 9707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:53.100816 9707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:53.100870 9707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:53.101414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9640
I20260925 12:12:53.101518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:12:53.101496 9707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } }
I20260925 12:12:53.101589 9707 leader_election.cc:302] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 355c47f6f91a49b399eb6c9ed07fda78; no voters:
I20260925 12:12:53.101732 392 external_mini_cluster.cc:1428] Setting key c312af83c3455c0dbf90e0e02ac360d1
I20260925 12:12:53.101752 9707 leader_election.cc:288] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:12:53.101811 9710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:53.102008 9707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:53.102029 9710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [term 1 LEADER]: Becoming Leader. State: Replica: 355c47f6f91a49b399eb6c9ed07fda78, State: Running, Role: LEADER
I20260925 12:12:53.102216 9710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } }
I20260925 12:12:53.102617 9711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "355c47f6f91a49b399eb6c9ed07fda78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } } }
I20260925 12:12:53.102694 9711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:53.102898 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355c47f6f91a49b399eb6c9ed07fda78. Latest consensus state: current_term: 1 leader_uuid: "355c47f6f91a49b399eb6c9ed07fda78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355c47f6f91a49b399eb6c9ed07fda78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39597 } } }
I20260925 12:12:53.102924 9721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:53.103018 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:53.103560 9721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:53.107779 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.102348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.62:39597)
0925 12:12:53.102565 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.102574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.102646 (+ 72us) server_negotiation.cc:407] Connection header received
0925 12:12:53.102851 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.102860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.102965 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.103123 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.103485 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.104837 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.105604 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.106291 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.106418 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.107102 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.107106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.107107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.107307 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.107348 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.107362 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.107432 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.107498 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.107574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260925 12:12:53.108042 9721 catalog_manager.cc:1347] Generated new cluster ID: 8503c66a943e403aaa57f6b547c204d8
I20260925 12:12:53.108088 9721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:12:53.111970 9721 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:12:53.112654 9721 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:53.124442 9721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355c47f6f91a49b399eb6c9ed07fda78: Generated new TSK 0
I20260925 12:12:53.124672 9721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:12:53.125808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--builtin_ntp_servers=127.0.98.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 12:12:53.218478 9731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.218719 9731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.218755 9731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.219895 9731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.219977 9731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.220021 9731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:53.221516 9731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.221789 9731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.222093 9731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:12:53.228006 9731 server_base.cc:1034] running on GCE node
W20260925 12:12:53.227865 9737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.227878 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.227870 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.228437 9731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.228735 9731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.229918 9731 hybrid_clock.cc:648] HybridClock initialized: now 1790338373229904 us; error 28 us; skew 500 ppm
I20260925 12:12:53.231309 9731 webserver.cc:460] Webserver started at http://127.0.98.1:38677/ using document root <none> and password file <none>
I20260925 12:12:53.231582 9731 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.231654 9731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.231792 9731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.233223 9731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "e00dcdb0c0b644588d914efccf753c47"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26294dc9202b31997f4eae9a353893be"
server_key_iv: "66df668c4527a6193c15e92ce18a8c89"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.233669 9731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e00dcdb0c0b644588d914efccf753c47"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26294dc9202b31997f4eae9a353893be"
server_key_iv: "66df668c4527a6193c15e92ce18a8c89"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.235314 9731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:53.236174 9745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.236366 9731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:53.236466 9731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "e00dcdb0c0b644588d914efccf753c47"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26294dc9202b31997f4eae9a353893be"
server_key_iv: "66df668c4527a6193c15e92ce18a8c89"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.236568 9731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.258550 9731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.258895 9731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.259089 9731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.259337 9731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:53.259724 9731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:53.259780 9731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.259811 9731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:53.259836 9731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.265811 9731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40169
I20260925 12:12:53.265883 9858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40169 every 8 connection(s)
I20260925 12:12:53.266217 9731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:53.271167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9731
I20260925 12:12:53.271241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:12:53.271466 392 external_mini_cluster.cc:1428] Setting key 0c0367e30a011bb3556484b01f12b994
I20260925 12:12:53.271693 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.266942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51789 (local address 127.0.98.62:39597)
0925 12:12:53.266994 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.266996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.267287 (+ 291us) server_negotiation.cc:407] Connection header received
0925 12:12:53.267848 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.267849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.267863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.267910 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.268618 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.269177 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.270212 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.270400 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.270475 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.271358 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.271359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.271359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.271364 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.271377 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.271381 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.271443 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.271586 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.271619 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:12:53.271898 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.266935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39597 (local address 127.0.98.1:51789)
0925 12:12:53.267153 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:53.267197 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:53.267730 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:53.267981 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:53.267985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:53.267995 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:53.268533 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.268539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.269268 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.269269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.270114 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.270116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.270483 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.270483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.270613 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.271159 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:53.271159 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:53.271187 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:53.271226 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:53.271465 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:53.271470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:53.271473 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:53.271551 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:12:53.271671 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260925 12:12:53.272326 9859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39597
I20260925 12:12:53.272431 9859 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:53.272583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--builtin_ntp_servers=127.0.98.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 12:12:53.272641 9859 heartbeater.cc:511] Master 127.0.98.62:39597 requested a full tablet report, sending...
I20260925 12:12:53.273221 9671 ts_manager.cc:194] Registered new tserver with Master: e00dcdb0c0b644588d914efccf753c47 (127.0.98.1:40169)
I20260925 12:12:53.274076 9671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51789
W20260925 12:12:53.353535 9862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.353785 9862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.353818 9862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.354959 9862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.355036 9862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.355083 9862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:53.356438 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.356735 9862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.357043 9862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.362140 9868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.362124 9867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.362159 9870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.362290 9862 server_base.cc:1034] running on GCE node
I20260925 12:12:53.362551 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.362753 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.363888 9862 hybrid_clock.cc:648] HybridClock initialized: now 1790338373363876 us; error 28 us; skew 500 ppm
I20260925 12:12:53.365197 9862 webserver.cc:460] Webserver started at http://127.0.98.2:40817/ using document root <none> and password file <none>
I20260925 12:12:53.365463 9862 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.365510 9862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.365633 9862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.366686 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "847701ba0f564d2d86859fe93721aebe"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "ed7ce1481da905901dd3a99369877f81"
server_key_iv: "c3686aa68d6bb4ef8eda36d1a6499624"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.367057 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "847701ba0f564d2d86859fe93721aebe"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "ed7ce1481da905901dd3a99369877f81"
server_key_iv: "c3686aa68d6bb4ef8eda36d1a6499624"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.368309 9862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:12:53.369145 9876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.369328 9862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:12:53.369400 9862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "847701ba0f564d2d86859fe93721aebe"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "ed7ce1481da905901dd3a99369877f81"
server_key_iv: "c3686aa68d6bb4ef8eda36d1a6499624"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.369480 9862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.380230 9862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.380640 9862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.380798 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.381031 9862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:53.381356 9862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:53.381387 9862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.381410 9862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:53.381464 9862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.387279 9862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46545
I20260925 12:12:53.387351 9989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46545 every 8 connection(s)
I20260925 12:12:53.387686 9862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:12:53.392596 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.388450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38473 (local address 127.0.98.62:39597)
0925 12:12:53.388521 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.388523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.388773 (+ 250us) server_negotiation.cc:407] Connection header received
0925 12:12:53.389247 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.389248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.389258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.389309 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.389916 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.390400 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.391330 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.391516 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.391567 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.392312 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.392313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.392314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.392318 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.392331 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.392335 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.392377 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.392465 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.392522 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:12:53.392839 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.388423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39597 (local address 127.0.98.2:38473)
0925 12:12:53.388642 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:53.388675 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:53.389163 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:53.389368 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:53.389372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:53.389379 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:53.389849 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.389853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.390481 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.390482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.391257 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.391258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.391590 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.391591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.391697 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.392185 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:53.392186 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:53.392214 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:53.392236 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:53.392389 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:53.392392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:53.392394 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:53.392444 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:12:53.392590 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260925 12:12:53.393301 9990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39597
I20260925 12:12:53.393376 9990 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:53.393517 9990 heartbeater.cc:511] Master 127.0.98.62:39597 requested a full tablet report, sending...
I20260925 12:12:53.393815 9671 ts_manager.cc:194] Registered new tserver with Master: 847701ba0f564d2d86859fe93721aebe (127.0.98.2:46545)
I20260925 12:12:53.394199 9671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38473
I20260925 12:12:53.397570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9862
I20260925 12:12:53.397673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:53.397871 392 external_mini_cluster.cc:1428] Setting key c756cb6237832fba37f983b943ad55ab
I20260925 12:12:53.398725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--builtin_ntp_servers=127.0.98.20:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 12:12:53.482358 9993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.482568 9993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.482589 9993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.483649 9993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.483681 9993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.483706 9993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:53.485041 9993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.485231 9993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.485484 9993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.490319 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.490337 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.490377 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.490509 9993 server_base.cc:1034] running on GCE node
I20260925 12:12:53.490798 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.490988 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.492125 9993 hybrid_clock.cc:648] HybridClock initialized: now 1790338373492124 us; error 37 us; skew 500 ppm
I20260925 12:12:53.493402 9993 webserver.cc:460] Webserver started at http://127.0.98.3:45705/ using document root <none> and password file <none>
I20260925 12:12:53.493651 9993 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.493696 9993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.493845 9993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.494884 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2d92fd7dd6d410291840355656fccc7"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "0a7e29986b08e4afb5844bfc749f05b4"
server_key_iv: "a7212ad97e64fc2b65263561d7117d10"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.495229 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d92fd7dd6d410291840355656fccc7"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "0a7e29986b08e4afb5844bfc749f05b4"
server_key_iv: "a7212ad97e64fc2b65263561d7117d10"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.496560 9993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:12:53.497344 10007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.497514 9993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:53.497610 9993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "f2d92fd7dd6d410291840355656fccc7"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "0a7e29986b08e4afb5844bfc749f05b4"
server_key_iv: "a7212ad97e64fc2b65263561d7117d10"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.497692 9993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.507586 9993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.507896 9993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.508054 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.508271 9993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:53.508632 9993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:53.508683 9993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.508724 9993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:53.508751 9993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.514636 9993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41065
I20260925 12:12:53.514705 10120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41065 every 8 connection(s)
I20260925 12:12:53.515070 9993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:12:53.520797 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.516390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33591 (local address 127.0.98.62:39597)
0925 12:12:53.516471 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.516473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.516762 (+ 289us) server_negotiation.cc:407] Connection header received
0925 12:12:53.517407 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.517408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.517419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.517485 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.518144 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.518635 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.519609 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.519792 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.519848 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.520604 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.520605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.520605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.520609 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.520622 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.520625 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.520668 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.520717 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.520739 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:12:53.521044 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.516066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39597 (local address 127.0.98.3:33591)
0925 12:12:53.516638 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:53.516683 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:53.517328 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:53.517542 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:53.517545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:53.517553 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:53.518066 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.518070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.518718 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.518719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.519528 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:53.519530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:53.519866 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:53.519866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:53.519970 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:53.520441 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:53.520441 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:53.520469 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:53.520528 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:53.520668 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:53.520671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:53.520673 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:53.520719 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:12:53.520789 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":100,"threads_started":1}
I20260925 12:12:53.521476 10121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39597
I20260925 12:12:53.521557 10121 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:53.521688 10121 heartbeater.cc:511] Master 127.0.98.62:39597 requested a full tablet report, sending...
I20260925 12:12:53.521998 9671 ts_manager.cc:194] Registered new tserver with Master: f2d92fd7dd6d410291840355656fccc7 (127.0.98.3:41065)
I20260925 12:12:53.522369 9671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33591
I20260925 12:12:53.523267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 9993
I20260925 12:12:53.523365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:53.523579 392 external_mini_cluster.cc:1428] Setting key 205403b24122ce859fae61d65eb52f9e
I20260925 12:12:53.524315 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 12:12:53.525969 9714 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:53.525558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.0.98.62:39597)
0925 12:12:53.525593 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.525595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.525696 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:12:53.525734 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.525735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.525744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.525790 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.525894 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50486: BlockingRecv error: recv got EOF from 127.0.0.1:50486 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:12:53.526077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9731
I20260925 12:12:53.531765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9862
I20260925 12:12:53.535950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9993
I20260925 12:12:53.540124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 9640
2026-09-25T12:12:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (609 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T12:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:53Z Disabled control of system clock
I20260925 12:12:53.565936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:12:53.645699 10132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.645908 10132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.645929 10132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.647056 10132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:53.647094 10132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.647106 10132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.647119 10132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:53.647130 10132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:53.648411 10132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.648634 10132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.648886 10132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.653728 10138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.653724 10140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.653882 10132 server_base.cc:1034] running on GCE node
W20260925 12:12:53.653748 10137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.654206 10132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.654507 10132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.655678 10132 hybrid_clock.cc:648] HybridClock initialized: now 1790338373655668 us; error 39 us; skew 500 ppm
I20260925 12:12:53.656973 10132 webserver.cc:460] Webserver started at http://127.0.98.62:32971/ using document root <none> and password file <none>
I20260925 12:12:53.657236 10132 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.657315 10132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.657482 10132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.658638 10132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "79bae1a85e9549c9ace4adfdbea6f3a3"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "98f37b663c84f1b4e04f62468050c2d4"
server_key_iv: "db554f5282a7ab1b02a41dd507f385f6"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.658989 10132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "79bae1a85e9549c9ace4adfdbea6f3a3"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "98f37b663c84f1b4e04f62468050c2d4"
server_key_iv: "db554f5282a7ab1b02a41dd507f385f6"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.660359 10132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:53.661204 10146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.661406 10132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:53.661509 10132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "79bae1a85e9549c9ace4adfdbea6f3a3"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "98f37b663c84f1b4e04f62468050c2d4"
server_key_iv: "db554f5282a7ab1b02a41dd507f385f6"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.661583 10132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.664939 10132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.665203 10132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.665349 10132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.669039 10132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42869
I20260925 12:12:53.669081 10198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42869 every 8 connection(s)
I20260925 12:12:53.669433 10132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:53.669750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10132
I20260925 12:12:53.669917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:12:53.670149 392 external_mini_cluster.cc:1428] Setting key b2d9514c16aedb9eca65486caa7ae8fe
I20260925 12:12:53.670192 10199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:53.673156 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.670887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.98.62:42869)
0925 12:12:53.671084 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.671092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.671123 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:12:53.671776 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.671785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.671841 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.672003 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.672736 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.672740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.672740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.672744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.672782 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.672803 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.672865 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.672978 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.673021 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260925 12:12:53.673758 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3: Bootstrap starting.
I20260925 12:12:53.674358 10199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:53.674592 10199 log.cc:824] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:53.675230 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3: No bootstrap required, opened a new log
I20260925 12:12:53.676743 10199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260925 12:12:53.676874 10199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:53.676896 10199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bae1a85e9549c9ace4adfdbea6f3a3, State: Initialized, Role: FOLLOWER
I20260925 12:12:53.677009 10199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260925 12:12:53.677098 10199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:53.677119 10199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:53.677146 10199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:53.677677 10199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260925 12:12:53.677762 10199 leader_election.cc:302] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79bae1a85e9549c9ace4adfdbea6f3a3; no voters:
I20260925 12:12:53.677903 10199 leader_election.cc:288] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:12:53.678155 10199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:53.678160 10204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:53.678311 10204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 79bae1a85e9549c9ace4adfdbea6f3a3, State: Running, Role: LEADER
I20260925 12:12:53.678414 10204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260925 12:12:53.678807 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260925 12:12:53.678892 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:53.679047 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79bae1a85e9549c9ace4adfdbea6f3a3. Latest consensus state: current_term: 1 leader_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79bae1a85e9549c9ace4adfdbea6f3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260925 12:12:53.679092 10213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:53.679111 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:53.679721 10213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:53.681247 10213 catalog_manager.cc:1347] Generated new cluster ID: d2e1aa0ffaa8469c9d4fd272db366502
I20260925 12:12:53.681300 10213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:12:53.686693 10213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:12:53.686875 10213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:53.690652 10213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79bae1a85e9549c9ace4adfdbea6f3a3: Generated new TSK 0
I20260925 12:12:53.690826 10213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:12:53.698540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--builtin_ntp_servers=127.0.98.20:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:12:53.782217 10223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.782418 10223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.782441 10223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.783612 10223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.783653 10223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.783679 10223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:53.785065 10223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.785292 10223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.785622 10223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.790647 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.790658 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.790658 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.790932 10223 server_base.cc:1034] running on GCE node
I20260925 12:12:53.791136 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.791380 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.792549 10223 hybrid_clock.cc:648] HybridClock initialized: now 1790338373792532 us; error 30 us; skew 500 ppm
I20260925 12:12:53.793807 10223 webserver.cc:460] Webserver started at http://127.0.98.1:37037/ using document root <none> and password file <none>
I20260925 12:12:53.794055 10223 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.794126 10223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.794270 10223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.795318 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9661df0ddd24816af423a9d47324634"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "9fc182f68cdb028f03a1d258c88ff963"
server_key_iv: "1b8ec1a52e7679d13860c3159ffe8230"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.795683 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9661df0ddd24816af423a9d47324634"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "9fc182f68cdb028f03a1d258c88ff963"
server_key_iv: "1b8ec1a52e7679d13860c3159ffe8230"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.797183 10223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:53.798066 10237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.798273 10223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:53.798390 10223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "b9661df0ddd24816af423a9d47324634"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "9fc182f68cdb028f03a1d258c88ff963"
server_key_iv: "1b8ec1a52e7679d13860c3159ffe8230"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.798471 10223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.801831 10223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.802094 10223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.802229 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.802448 10223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:53.802744 10223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:53.802795 10223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.802825 10223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:53.802851 10223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.808700 10223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36893
I20260925 12:12:53.808780 10350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36893 every 8 connection(s)
I20260925 12:12:53.809093 10223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:53.811976 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.809801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59159 (local address 127.0.98.62:42869)
0925 12:12:53.809849 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.809851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.810162 (+ 311us) server_negotiation.cc:407] Connection header received
0925 12:12:53.810597 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.810598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.810602 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.810657 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.811799 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.811800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.811800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.811803 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.811813 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.811816 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.811851 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.811916 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.811928 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:12:53.812152 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.809800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:59159)
0925 12:12:53.810037 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:53.810071 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:53.810519 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:53.810710 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:53.810713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:53.810721 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:53.811666 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:53.811666 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:53.811694 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:53.811721 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:53.811853 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:53.811855 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:53.811857 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:53.811910 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:12:53.811956 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":127,"threads_started":1}
I20260925 12:12:53.812666 10351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260925 12:12:53.812769 10351 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:53.812927 10351 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260925 12:12:53.813414 10163 ts_manager.cc:194] Registered new tserver with Master: b9661df0ddd24816af423a9d47324634 (127.0.98.1:36893)
I20260925 12:12:53.813544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10223
I20260925 12:12:53.813647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:12:53.813856 392 external_mini_cluster.cc:1428] Setting key b5eba8dca6f128a5298bf872e2a5d349
I20260925 12:12:53.814724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--builtin_ntp_servers=127.0.98.20:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:12:53.894222 10354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:53.894469 10354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:53.894503 10354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:53.895689 10354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:53.895756 10354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:53.895797 10354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:53.897143 10354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:53.897418 10354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:53.897708 10354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:53.902709 10362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.902834 10360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:53.902716 10359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:53.902938 10354 server_base.cc:1034] running on GCE node
I20260925 12:12:53.903263 10354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:53.903518 10354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:53.904675 10354 hybrid_clock.cc:648] HybridClock initialized: now 1790338373904662 us; error 44 us; skew 500 ppm
I20260925 12:12:53.905964 10354 webserver.cc:460] Webserver started at http://127.0.98.2:34879/ using document root <none> and password file <none>
I20260925 12:12:53.906208 10354 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:53.906276 10354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:53.906421 10354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:53.907475 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "c825f5d2c90343d487b9cfa751963223"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26e75ee4984fdf81ca3376bf73b98c8a"
server_key_iv: "f48564af80941b98e22c46d036f10844"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.907840 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c825f5d2c90343d487b9cfa751963223"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26e75ee4984fdf81ca3376bf73b98c8a"
server_key_iv: "f48564af80941b98e22c46d036f10844"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.909233 10354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:12:53.910058 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.910228 10354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:53.910320 10354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "c825f5d2c90343d487b9cfa751963223"
format_stamp: "Formatted at 2026-09-25 12:12:53 on dist-test-slave-2hqq"
server_key: "26e75ee4984fdf81ca3376bf73b98c8a"
server_key_iv: "f48564af80941b98e22c46d036f10844"
server_key_version: "encryptionkey@0"
I20260925 12:12:53.910418 10354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:53.913774 10354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:53.914130 10354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:53.914304 10354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:53.914520 10354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:53.914826 10354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:53.914881 10354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.914920 10354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:53.914947 10354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:53.920753 10354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35915
I20260925 12:12:53.920823 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35915 every 8 connection(s)
I20260925 12:12:53.921149 10354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:12:53.923959 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36703 (local address 127.0.98.62:42869)
0925 12:12:53.921950 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:53.921952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:53.922152 (+ 200us) server_negotiation.cc:407] Connection header received
0925 12:12:53.922582 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:53.922583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:53.922588 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:53.922648 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:53.923782 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:53.923783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:53.923783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:53.923785 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:53.923794 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:53.923797 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:53.923836 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:53.923896 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 12:12:53.923907 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 12:12:53.924168 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:53.921835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:36703)
0925 12:12:53.922022 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:53.922061 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:53.922503 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:53.922697 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:53.922700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:53.922707 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:53.923657 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:53.923658 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:53.923685 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:53.923713 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:53.923837 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:53.923839 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:53.923841 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:53.923894 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:12:53.923939 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 12:12:53.924698 10482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260925 12:12:53.924773 10482 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:53.924933 10482 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260925 12:12:53.925305 10163 ts_manager.cc:194] Registered new tserver with Master: c825f5d2c90343d487b9cfa751963223 (127.0.98.2:35915)
I20260925 12:12:53.929315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10354
I20260925 12:12:53.929423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:53.929644 392 external_mini_cluster.cc:1428] Setting key 0ccd74ceb265f5abe0195c955993a6a0
I20260925 12:12:53.930598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--builtin_ntp_servers=127.0.98.20:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 12:12:54.015712 10485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:54.015926 10485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:54.015950 10485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:54.017144 10485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:54.017184 10485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:54.017210 10485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:54.018561 10485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:54.018819 10485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:54.019137 10485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:54.024214 10491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:54.024237 10485 server_base.cc:1034] running on GCE node
W20260925 12:12:54.024209 10493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:54.024210 10490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:54.024647 10485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:54.024940 10485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:54.026103 10485 hybrid_clock.cc:648] HybridClock initialized: now 1790338374026076 us; error 50 us; skew 500 ppm
I20260925 12:12:54.027428 10485 webserver.cc:460] Webserver started at http://127.0.98.3:33273/ using document root <none> and password file <none>
I20260925 12:12:54.027678 10485 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:54.027724 10485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:54.027870 10485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:54.028978 10485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "f03e754573004567adb9af2738bc4497"
format_stamp: "Formatted at 2026-09-25 12:12:54 on dist-test-slave-2hqq"
server_key: "020846780bd99228472b5f5b6ca5be9d"
server_key_iv: "60210de9a06efbf5207ff7bde7c26e4b"
server_key_version: "encryptionkey@0"
I20260925 12:12:54.029356 10485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f03e754573004567adb9af2738bc4497"
format_stamp: "Formatted at 2026-09-25 12:12:54 on dist-test-slave-2hqq"
server_key: "020846780bd99228472b5f5b6ca5be9d"
server_key_iv: "60210de9a06efbf5207ff7bde7c26e4b"
server_key_version: "encryptionkey@0"
I20260925 12:12:54.030660 10485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:12:54.031394 10499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:54.031548 10485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:54.031633 10485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "f03e754573004567adb9af2738bc4497"
format_stamp: "Formatted at 2026-09-25 12:12:54 on dist-test-slave-2hqq"
server_key: "020846780bd99228472b5f5b6ca5be9d"
server_key_iv: "60210de9a06efbf5207ff7bde7c26e4b"
server_key_version: "encryptionkey@0"
I20260925 12:12:54.031685 10485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:54.035000 10485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:54.035207 10485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:54.035300 10485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:54.035447 10485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:54.035676 10485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:54.035705 10485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:54.035724 10485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:54.035737 10485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:54.041404 10485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34463
I20260925 12:12:54.041445 10612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34463 every 8 connection(s)
I20260925 12:12:54.041741 10485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:12:54.044811 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:54.042466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45159 (local address 127.0.98.62:42869)
0925 12:12:54.042528 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:54.042530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:54.042797 (+ 267us) server_negotiation.cc:407] Connection header received
0925 12:12:54.043226 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:54.043228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:54.043233 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:54.043287 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:54.044637 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:54.044639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:54.044639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:12:54.044642 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:54.044655 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:12:54.044658 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:54.044704 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:54.044749 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 12:12:54.044760 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:12:54.045053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10485
I20260925 12:12:54.045077 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:54.042457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:45159)
0925 12:12:54.042673 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:54.042705 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:54.043147 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:54.043358 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:54.043362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:54.043370 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:54.044467 (+ 1097us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:12:54.044467 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:54.044535 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:12:54.044563 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:54.044703 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:54.044705 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:54.044707 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:12:54.044764 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:12:54.044809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260925 12:12:54.045164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:54.045388 392 external_mini_cluster.cc:1428] Setting key 28226c5221f3b8026d017571468f94b7
I20260925 12:12:54.045639 10613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260925 12:12:54.045784 10613 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:54.045938 10613 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260925 12:12:54.046281 10163 ts_manager.cc:194] Registered new tserver with Master: f03e754573004567adb9af2738bc4497 (127.0.98.3:34463)
I20260925 12:12:54.057291 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 12:12:54.059219 10202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:54.058778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.98.62:42869)
0925 12:12:54.058818 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:54.058820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:54.058877 (+ 57us) server_negotiation.cc:407] Connection header received
0925 12:12:54.058947 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:54.058947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:54.058951 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:54.059007 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:54.059137 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56020: BlockingRecv error: recv got EOF from 127.0.0.1:56020 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:12:54.059329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10223
I20260925 12:12:54.064011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10354
I20260925 12:12:54.068287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10485
I20260925 12:12:54.072602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10132
2026-09-25T12:12:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (533 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:12:54 dist-test-slave-2hqq krb5kdc[10623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:12:54 dist-test-slave-2hqq krb5kdc[10623](info): set up 2 sockets
Sep 25 12:12:54 dist-test-slave-2hqq krb5kdc[10623](info): commencing operation
krb5kdc: starting...
W20260925 12:12:55.120847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:12:55.236775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:55.316205 10636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:55.316404 10636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:55.316426 10636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:55.317605 10636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:12:55.317646 10636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:55.317660 10636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:55.317710 10636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:12:55.317732 10636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:12:55.319182 10636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:55.319430 10636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:55.319662 10636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:55.324456 10644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.324620 10636 server_base.cc:1034] running on GCE node
W20260925 12:12:55.324524 10642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:55.324470 10641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.325006 10636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:55.325241 10636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:55.326381 10636 hybrid_clock.cc:648] HybridClock initialized: now 1790338375326368 us; error 28 us; skew 500 ppm
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:55.334261 10636 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:12:55.334640 10636 webserver.cc:460] Webserver started at http://127.0.98.62:40191/ using document root <none> and password file <none>
I20260925 12:12:55.334832 10636 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:55.334873 10636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:55.334959 10636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:55.335932 10636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "2e9b37d9e0dc438bbdb4e1705c46f280"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "e838d0312a2f94e4c7d1f0156ba059af"
server_key_iv: "9d729882054af10e4290cdd1cb1b9ba9"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.336216 10636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e9b37d9e0dc438bbdb4e1705c46f280"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "e838d0312a2f94e4c7d1f0156ba059af"
server_key_iv: "9d729882054af10e4290cdd1cb1b9ba9"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.337543 10636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:12:55.338267 10651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.338459 10636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:12:55.338529 10636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "2e9b37d9e0dc438bbdb4e1705c46f280"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "e838d0312a2f94e4c7d1f0156ba059af"
server_key_iv: "9d729882054af10e4290cdd1cb1b9ba9"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.338619 10636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:55.356942 10636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:55.358774 10636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:55.358932 10636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:55.362958 10636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41999
I20260925 12:12:55.363071 10703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41999 every 8 connection(s)
I20260925 12:12:55.363365 10636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:12:55.364248 10704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:55.366521 10704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: Bootstrap starting.
I20260925 12:12:55.367094 10704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:55.367393 10704 log.cc:824] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:55.368108 10704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: No bootstrap required, opened a new log
I20260925 12:12:55.369673 10704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } }
I20260925 12:12:55.369808 10704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:55.369831 10704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9b37d9e0dc438bbdb4e1705c46f280, State: Initialized, Role: FOLLOWER
I20260925 12:12:55.370007 10704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } }
I20260925 12:12:55.370067 10704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:12:55.370088 10704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:12:55.370157 10704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:55.370916 10704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } }
I20260925 12:12:55.371030 10704 leader_election.cc:302] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e9b37d9e0dc438bbdb4e1705c46f280; no voters:
I20260925 12:12:55.371243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10636
I20260925 12:12:55.371258 10704 leader_election.cc:288] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:12:55.371287 10707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:55.371342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:12:55.371389 10707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [term 1 LEADER]: Becoming Leader. State: Replica: 2e9b37d9e0dc438bbdb4e1705c46f280, State: Running, Role: LEADER
I20260925 12:12:55.371512 10704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:12:55.371546 392 external_mini_cluster.cc:1428] Setting key c212fa1b0005beceedfbda3f418a7385
I20260925 12:12:55.371520 10707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } }
I20260925 12:12:55.371948 10708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e9b37d9e0dc438bbdb4e1705c46f280. Latest consensus state: current_term: 1 leader_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } } }
I20260925 12:12:55.372113 10708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:55.371959 10709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9b37d9e0dc438bbdb4e1705c46f280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41999 } } }
I20260925 12:12:55.372355 10709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280 [sys.catalog]: This master's current role is: LEADER
I20260925 12:12:55.372416 10718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:12:55.373018 10718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:12:55.374647 10718 catalog_manager.cc:1347] Generated new cluster ID: 680dc4993130491ca331232c140ec31f
I20260925 12:12:55.374706 10718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:55.382340 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.372096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.98.62:41999)
0925 12:12:55.372259 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.372268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.372299 (+ 31us) server_negotiation.cc:407] Connection header received
0925 12:12:55.372415 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.372423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.372482 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.373328 (+ 846us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:55.373571 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.374985 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.375733 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.376153 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.376287 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.378530 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:55.378534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:55.378534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:55.378553 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:55.380739 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.381129 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.381132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.381133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.381228 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.381452 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.381453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.381453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.381689 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:55.381887 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.381952 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.382181 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 12:12:55.393537 10718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:12:55.394068 10718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:12:55.397616 10718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: Generated new TSK 0
I20260925 12:12:55.397794 10718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:12:55.429910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:55.509662 10730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:55.509919 10730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:55.509955 10730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:55.511135 10730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:55.511219 10730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:55.511265 10730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:12:55.512753 10730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:55.513010 10730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:55.513242 10730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:55.518488 10736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.518586 10730 server_base.cc:1034] running on GCE node
W20260925 12:12:55.518527 10738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:55.518476 10735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.518893 10730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:55.519142 10730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:55.520308 10730 hybrid_clock.cc:648] HybridClock initialized: now 1790338375520291 us; error 28 us; skew 500 ppm
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:55.523015 10730 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:12:55.523561 10730 webserver.cc:460] Webserver started at http://127.0.98.1:34009/ using document root <none> and password file <none>
I20260925 12:12:55.523813 10730 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:55.523877 10730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:55.524057 10730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:55.525364 10730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "366bddf13dc4aecad167c8754cb800fe"
server_key_iv: "a2a012902eefea34a973c5f093b1fb44"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.525813 10730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "366bddf13dc4aecad167c8754cb800fe"
server_key_iv: "a2a012902eefea34a973c5f093b1fb44"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.527123 10730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:12:55.527855 10745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.528026 10730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:55.528126 10730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "366bddf13dc4aecad167c8754cb800fe"
server_key_iv: "a2a012902eefea34a973c5f093b1fb44"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.528225 10730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:55.542977 10730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:55.544965 10730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:55.545121 10730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:55.545351 10730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:55.545648 10730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:55.545702 10730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.545735 10730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:55.545763 10730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.551695 10730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41357
I20260925 12:12:55.551749 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41357 every 8 connection(s)
I20260925 12:12:55.552045 10730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:12:55.554924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10730
I20260925 12:12:55.555039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:12:55.555250 392 external_mini_cluster.cc:1428] Setting key 1c41f7db17ee84e0fb4de25f66922ad4
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:55.561762 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.552764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60351 (local address 127.0.98.62:41999)
0925 12:12:55.552844 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.552846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.553089 (+ 243us) server_negotiation.cc:407] Connection header received
0925 12:12:55.553560 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.553561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.553574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.553625 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:55.554533 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.555025 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.556290 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.556507 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.556586 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.558940 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:55.558942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:55.558942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:55.558950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:55.560352 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.560802 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.560803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.560804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.560863 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.561111 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.561112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.561112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.561230 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:55.561306 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.561575 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.561659 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 12:12:55.561990 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.552742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41999 (local address 127.0.98.1:60351)
0925 12:12:55.552949 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.552993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.553477 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.553679 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.553682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.554255 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:55.554443 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.554447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.555157 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.555158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.556188 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.556191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.556592 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.556593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.556724 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.557252 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:55.557263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:55.558846 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:55.560437 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.560443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.560446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.560714 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.560931 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.560932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.560932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.561044 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.561297 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:55.561303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:55.561445 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.561553 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.561706 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260925 12:12:55.562474 10859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41999
I20260925 12:12:55.562563 10859 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:55.562744 10859 heartbeater.cc:511] Master 127.0.98.62:41999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:12:55.563261 10668 ts_manager.cc:194] Registered new tserver with Master: 810ec5e7a7ca4e5d9eb11cfb4355d27e (127.0.98.1:41357)
I20260925 12:12:55.563876 10668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:12:55.581437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:55.660670 10864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:55.660907 10864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:55.660948 10864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:55.662097 10864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:55.662168 10864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:55.662222 10864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:12:55.663655 10864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:55.663923 10864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:55.664158 10864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:55.669039 10872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:55.669039 10870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:55.669082 10869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.669374 10864 server_base.cc:1034] running on GCE node
I20260925 12:12:55.669580 10864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:55.669821 10864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:55.670946 10864 hybrid_clock.cc:648] HybridClock initialized: now 1790338375670924 us; error 37 us; skew 500 ppm
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:55.673575 10864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:12:55.673943 10864 webserver.cc:460] Webserver started at http://127.0.98.2:43959/ using document root <none> and password file <none>
I20260925 12:12:55.674118 10864 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:55.674157 10864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:55.674263 10864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:55.675447 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "986db6229294428dbfa6c776fd9d51e9"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "ccc90c48d7aec3b20a4fcc494c501006"
server_key_iv: "801ee7e6c35952dbf602024af515626c"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.675756 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "986db6229294428dbfa6c776fd9d51e9"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "ccc90c48d7aec3b20a4fcc494c501006"
server_key_iv: "801ee7e6c35952dbf602024af515626c"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.677059 10864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 12:12:55.677747 10879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.677918 10864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:12:55.678004 10864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "986db6229294428dbfa6c776fd9d51e9"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "ccc90c48d7aec3b20a4fcc494c501006"
server_key_iv: "801ee7e6c35952dbf602024af515626c"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.678056 10864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:55.704108 10864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:55.706213 10864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:55.706326 10864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:55.706516 10864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:55.706837 10864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:55.706871 10864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.706893 10864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:55.706939 10864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.712791 10864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37727
I20260925 12:12:55.712852 10992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37727 every 8 connection(s)
I20260925 12:12:55.713184 10864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:12:55.716030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10864
I20260925 12:12:55.716152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:12:55.716351 392 external_mini_cluster.cc:1428] Setting key e6e32662fd84e9982065e663667a3a2c
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:55.723850 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.713935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41571 (local address 127.0.98.62:41999)
0925 12:12:55.713982 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.713984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.714173 (+ 189us) server_negotiation.cc:407] Connection header received
0925 12:12:55.714759 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.714759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.714771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.714824 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:55.715828 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.716446 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.717953 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.718138 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.718207 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.720539 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:55.720540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:55.720541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:55.720555 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:55.722436 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.722885 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.722887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.722887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.722943 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.723200 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.723201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.723201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.723326 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:55.723399 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.723669 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.723759 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:12:55.724040 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.713891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41999 (local address 127.0.98.2:41571)
0925 12:12:55.714028 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.714074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.714636 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.714895 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.714898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.715554 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:55.715748 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.715752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.716581 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.716582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.717857 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.717860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.718218 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.718219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.718369 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.718937 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:55.718953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:55.720403 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:55.722545 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.722553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.722556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.722790 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.723024 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.723025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.723025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.723131 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.723390 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:55.723397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:55.723540 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.723650 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.723807 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260925 12:12:55.724567 10993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41999
I20260925 12:12:55.724663 10993 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:12:55.724890 10993 heartbeater.cc:511] Master 127.0.98.62:41999 requested a full tablet report, sending...
I20260925 12:12:55.725332 10668 ts_manager.cc:194] Registered new tserver with Master: 986db6229294428dbfa6c776fd9d51e9 (127.0.98.2:37727)
I20260925 12:12:55.725716 10668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:12:55.743358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:12:55.823025 10998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:12:55.823226 10998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:12:55.823249 10998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:12:55.824386 10998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:12:55.824424 10998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:12:55.824481 10998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:12:55.825944 10998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:12:55.826180 10998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:12:55.826402 10998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:12:55.831425 11003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:12:55.831432 11004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.831472 10998 server_base.cc:1034] running on GCE node
W20260925 12:12:55.831423 11006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:12:55.831811 10998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:12:55.832047 10998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:12:55.833230 10998 hybrid_clock.cc:648] HybridClock initialized: now 1790338375833209 us; error 32 us; skew 500 ppm
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:12:55.835553 10998 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:12:55.836017 10998 webserver.cc:460] Webserver started at http://127.0.98.3:46339/ using document root <none> and password file <none>
I20260925 12:12:55.836242 10998 fs_manager.cc:359] Metadata directory not provided
I20260925 12:12:55.836311 10998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:12:55.836458 10998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:12:55.837535 10998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "35fcdfc1094c43bbab49797d12032dc3"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "70935fa780c1f32392b77b1820b7bcad"
server_key_iv: "1d60ce90ca31dd52ecf4e8f60467926d"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.837929 10998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35fcdfc1094c43bbab49797d12032dc3"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "70935fa780c1f32392b77b1820b7bcad"
server_key_iv: "1d60ce90ca31dd52ecf4e8f60467926d"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.839294 10998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:12:55.839973 11013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.840140 10998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:12:55.840234 10998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "35fcdfc1094c43bbab49797d12032dc3"
format_stamp: "Formatted at 2026-09-25 12:12:55 on dist-test-slave-2hqq"
server_key: "70935fa780c1f32392b77b1820b7bcad"
server_key_iv: "1d60ce90ca31dd52ecf4e8f60467926d"
server_key_version: "encryptionkey@0"
I20260925 12:12:55.840312 10998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:12:55.848209 10998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:12:55.850327 10998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:12:55.850488 10998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:12:55.850706 10998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:12:55.851009 10998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:12:55.851063 10998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.851104 10998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:12:55.851131 10998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:12:55.856945 10998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35453
I20260925 12:12:55.856987 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35453 every 8 connection(s)
I20260925 12:12:55.857283 10998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:12:55.857978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 10998
I20260925 12:12:55.858093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:12:55.858362 392 external_mini_cluster.cc:1428] Setting key 5ab9758daaebd909b89d51320a9d9687
Sep 25 12:12:55 dist-test-slave-2hqq krb5kdc[10623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:12:55.865985 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.858016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40359 (local address 127.0.98.62:41999)
0925 12:12:55.858108 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.858110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.858338 (+ 228us) server_negotiation.cc:407] Connection header received
0925 12:12:55.859027 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.859028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.859040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.859088 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:55.860048 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.860555 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.861511 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.861700 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.861754 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.863653 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:55.863654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:55.863655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:55.863660 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:55.864821 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.865173 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.865174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.865174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.865221 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.865422 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.865422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.865422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.865566 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:55.865637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.865861 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.865922 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 12:12:55.866114 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.857997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41999 (local address 127.0.98.3:40359)
0925 12:12:55.858195 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.858240 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.858923 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.859173 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.859177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.859791 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:12:55.859969 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.859973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.860646 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.860647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.861437 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.861439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.861773 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.861774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.861867 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.862281 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:12:55.862289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:12:55.863573 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:12:55.864887 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.864892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.864894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.865096 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.865276 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:12:55.865276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:12:55.865277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:12:55.865365 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:12:55.865630 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:12:55.865633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:12:55.865742 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.865824 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.865934 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260925 12:12:55.866526 11127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41999
I20260925 12:12:55.866613 11127 heartbeater.cc:464] Registering TS with master...
I20260925 12:12:55.866762 11127 heartbeater.cc:511] Master 127.0.98.62:41999 requested a full tablet report, sending...
I20260925 12:12:55.867117 10668 ts_manager.cc:194] Registered new tserver with Master: 35fcdfc1094c43bbab49797d12032dc3 (127.0.98.3:35453)
I20260925 12:12:55.867456 10668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40359
I20260925 12:12:55.870105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:12:55.877269 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.871434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.0.98.62:41999)
0925 12:12:55.871498 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.871500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.871592 (+ 92us) server_negotiation.cc:407] Connection header received
0925 12:12:55.871638 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.871639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.871649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.871694 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:12:55.872332 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.872890 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.873587 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.873775 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.873831 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.875168 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:12:55.875169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:12:55.875169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:12:55.875175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:12:55.876329 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.876675 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.876676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.876676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.876723 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:12:55.876873 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:12:55.876873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:12:55.876874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:12:55.876989 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:12:55.877055 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.877157 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.877212 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:12:55.878705 10668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:12:55.880545 10668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:12:55.886587 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.882584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41357 (local address 127.0.0.1:50164)
0925 12:12:55.882874 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.882883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.882921 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.883362 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.883365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.883368 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:55.883432 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.883433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884315 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.884316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.885446 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.885448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886309 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.886384 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886396 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.886467 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.886515 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":58,"threads_started":1}
I20260925 12:12:55.886600 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.883023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.0.98.1:41357)
0925 12:12:55.883190 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.883194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.883204 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:55.883247 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.883248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.883259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.883351 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:55.883515 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884215 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.885543 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886226 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886309 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886369 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.886482 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.886520 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260925 12:12:55.886790 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.882579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35453 (local address 127.0.0.1:46510)
0925 12:12:55.882686 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.882722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.882784 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.883460 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.883461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.883463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:55.883499 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.883501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884770 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.884771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.885691 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.885693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886528 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.886603 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886607 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.886669 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.886707 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260925 12:12:55.887087 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.882822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.3:35453)
0925 12:12:55.882996 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.883000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.883010 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:12:55.883052 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.883054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.883071 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.883464 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:55.883561 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884665 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.885772 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886378 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886446 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886495 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.886959 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.887011 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260925 12:12:55.887097 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.882601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.0.1:43780)
0925 12:12:55.883010 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.883021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.883070 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.883620 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.883621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.883623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:55.883676 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.883678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884892 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.884894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.885965 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.885967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886892 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.886955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886958 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.887005 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.887037 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260925 12:12:55.887374 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.882827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.0.98.2:37727)
0925 12:12:55.883220 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.883223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.883230 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:12:55.883258 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.883259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.883271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.883354 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:55.883760 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.884806 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886048 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.886744 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.886805 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.886857 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.887253 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.887295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":86,"threads_started":1}
I20260925 12:12:55.887640 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 (DEFAULT_TABLE table=test-table [id=146c979f16794103bf0a468abcaff510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:55.887692 11061 tablet_service.cc:1467] Processing CreateTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 (DEFAULT_TABLE table=test-table [id=146c979f16794103bf0a468abcaff510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:55.888005 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ca38a5fd3249899a7963d1fd12b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:55.888005 11061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ca38a5fd3249899a7963d1fd12b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:55.889171 10927 tablet_service.cc:1467] Processing CreateTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 (DEFAULT_TABLE table=test-table [id=146c979f16794103bf0a468abcaff510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:12:55.889531 10927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ca38a5fd3249899a7963d1fd12b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:12:55.891299 11142 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: Bootstrap starting.
I20260925 12:12:55.891552 11141 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Bootstrap starting.
I20260925 12:12:55.891963 11142 tablet_bootstrap.cc:654] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:55.892033 11141 tablet_bootstrap.cc:654] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:55.892551 11142 log.cc:824] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:55.892549 11141 log.cc:824] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:55.892664 11143 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Bootstrap starting.
I20260925 12:12:55.893170 11143 tablet_bootstrap.cc:654] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Neither blocks nor log segments found. Creating new log.
I20260925 12:12:55.893386 11142 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: No bootstrap required, opened a new log
I20260925 12:12:55.893465 11142 ts_tablet_manager.cc:1397] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:12:55.893705 11143 log.cc:824] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:12:55.894356 11141 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: No bootstrap required, opened a new log
I20260925 12:12:55.894403 11141 ts_tablet_manager.cc:1397] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 12:12:55.894440 11143 tablet_bootstrap.cc:492] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: No bootstrap required, opened a new log
I20260925 12:12:55.894498 11143 ts_tablet_manager.cc:1397] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:12:55.895524 11142 raft_consensus.cc:348] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.895689 11142 raft_consensus.cc:374] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:55.895614 11141 raft_consensus.cc:348] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.895722 11142 raft_consensus.cc:729] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810ec5e7a7ca4e5d9eb11cfb4355d27e, State: Initialized, Role: FOLLOWER
I20260925 12:12:55.895738 11141 raft_consensus.cc:374] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:55.895759 11141 raft_consensus.cc:729] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fcdfc1094c43bbab49797d12032dc3, State: Initialized, Role: FOLLOWER
I20260925 12:12:55.895843 11142 consensus_queue.cc:260] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.895870 11141 consensus_queue.cc:260] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.896097 11142 ts_tablet_manager.cc:1428] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:12:55.896153 11141 ts_tablet_manager.cc:1428] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:12:55.896138 10859 heartbeater.cc:503] Master 127.0.98.62:41999 was elected leader, sending a full tablet report...
I20260925 12:12:55.896262 11127 heartbeater.cc:503] Master 127.0.98.62:41999 was elected leader, sending a full tablet report...
I20260925 12:12:55.896404 11143 raft_consensus.cc:348] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.896582 11143 raft_consensus.cc:374] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:12:55.896624 11143 raft_consensus.cc:729] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986db6229294428dbfa6c776fd9d51e9, State: Initialized, Role: FOLLOWER
I20260925 12:12:55.896752 11143 consensus_queue.cc:260] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.897019 11143 ts_tablet_manager.cc:1428] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:12:55.897043 10993 heartbeater.cc:503] Master 127.0.98.62:41999 was elected leader, sending a full tablet report...
W20260925 12:12:55.963757 10994 tablet.cc:2369] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:12:55.984174 11149 raft_consensus.cc:482] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:12:55.984303 11149 raft_consensus.cc:504] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.984825 11149 leader_election.cc:288] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35fcdfc1094c43bbab49797d12032dc3 (127.0.98.3:35453), 810ec5e7a7ca4e5d9eb11cfb4355d27e (127.0.98.1:41357)
I20260925 12:12:55.987613 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.985009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41357 (local address 127.0.98.2:48643)
0925 12:12:55.985118 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.985132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.985183 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.985341 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.985342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.985343 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:55.985416 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.985418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.986049 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.986050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.986797 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.986799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.987352 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.987353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.987433 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.987436 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.987504 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.987545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 12:12:55.987613 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.985156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48643 (local address 127.0.98.1:41357)
0925 12:12:55.985212 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.985214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.985222 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:12:55.985249 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.985250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.985262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.985337 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:55.985489 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.985962 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.986893 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.987269 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.987352 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.987362 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.987510 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.987545 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:12:55.987852 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.985010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35453 (local address 127.0.98.2:52673)
0925 12:12:55.985288 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:55.985299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:55.985382 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:12:55.985529 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:12:55.985530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:12:55.985531 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:12:55.985600 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.985602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.986225 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.986227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.986982 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:12:55.986984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.987544 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.987545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:12:55.987614 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.987618 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:12:55.987703 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:12:55.987752 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":67,"thread_start_us":153,"threads_started":1}
I20260925 12:12:55.987943 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.985081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52673 (local address 127.0.98.3:35453)
0925 12:12:55.985139 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.985141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.985376 (+ 235us) server_negotiation.cc:407] Connection header received
0925 12:12:55.985439 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.985441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.985453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.985525 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:12:55.985685 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.986138 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.987078 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.987458 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:55.987542 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:55.987550 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:55.987825 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 12:12:55.987869 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:12:55.988001 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ca38a5fd3249899a7963d1fd12b6e5" candidate_uuid: "986db6229294428dbfa6c776fd9d51e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" is_pre_election: true
I20260925 12:12:55.988180 10813 raft_consensus.cc:2454] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986db6229294428dbfa6c776fd9d51e9 in term 0.
I20260925 12:12:55.988251 11081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ca38a5fd3249899a7963d1fd12b6e5" candidate_uuid: "986db6229294428dbfa6c776fd9d51e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fcdfc1094c43bbab49797d12032dc3" is_pre_election: true
I20260925 12:12:55.988413 11081 raft_consensus.cc:2454] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986db6229294428dbfa6c776fd9d51e9 in term 0.
I20260925 12:12:55.988449 10880 leader_election.cc:302] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810ec5e7a7ca4e5d9eb11cfb4355d27e, 986db6229294428dbfa6c776fd9d51e9; no voters:
I20260925 12:12:55.988587 11149 raft_consensus.cc:2781] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:12:55.988664 11149 raft_consensus.cc:482] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:12:55.988693 11149 raft_consensus.cc:3037] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:55.989415 11149 raft_consensus.cc:504] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.989566 11149 leader_election.cc:288] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [CANDIDATE]: Term 1 election: Requested vote from peers 35fcdfc1094c43bbab49797d12032dc3 (127.0.98.3:35453), 810ec5e7a7ca4e5d9eb11cfb4355d27e (127.0.98.1:41357)
I20260925 12:12:55.989758 11081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ca38a5fd3249899a7963d1fd12b6e5" candidate_uuid: "986db6229294428dbfa6c776fd9d51e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fcdfc1094c43bbab49797d12032dc3"
I20260925 12:12:55.989825 11081 raft_consensus.cc:3037] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:55.990531 11081 raft_consensus.cc:2454] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986db6229294428dbfa6c776fd9d51e9 in term 1.
I20260925 12:12:55.990649 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ca38a5fd3249899a7963d1fd12b6e5" candidate_uuid: "986db6229294428dbfa6c776fd9d51e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e"
I20260925 12:12:55.990707 10881 leader_election.cc:302] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fcdfc1094c43bbab49797d12032dc3, 986db6229294428dbfa6c776fd9d51e9; no voters:
I20260925 12:12:55.990759 10813 raft_consensus.cc:3037] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:12:55.990840 11149 raft_consensus.cc:2781] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:12:55.991011 11149 raft_consensus.cc:686] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 1 LEADER]: Becoming Leader. State: Replica: 986db6229294428dbfa6c776fd9d51e9, State: Running, Role: LEADER
I20260925 12:12:55.991120 11149 consensus_queue.cc:237] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } }
I20260925 12:12:55.991721 10813 raft_consensus.cc:2454] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986db6229294428dbfa6c776fd9d51e9 in term 1.
I20260925 12:12:55.992282 10667 catalog_manager.cc:5495] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 986db6229294428dbfa6c776fd9d51e9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "986db6229294428dbfa6c776fd9d51e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "986db6229294428dbfa6c776fd9d51e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260925 12:12:56.001489 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:12:55.999119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.0.98.2:37727)
0925 12:12:55.999170 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:12:55.999172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:12:55.999238 (+ 66us) server_negotiation.cc:407] Connection header received
0925 12:12:55.999302 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:12:55.999303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:12:55.999329 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:12:55.999389 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:12:55.999526 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:55.999995 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:56.000777 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:12:56.000971 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:12:56.001029 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:12:56.001159 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:12:56.001255 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:12:56.001316 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:12:56.001348 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:12:56.001364 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:12:56.003270 11081 raft_consensus.cc:1261] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 1 FOLLOWER]: Refusing update from remote peer 986db6229294428dbfa6c776fd9d51e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:56.003300 10813 raft_consensus.cc:1261] T 99ca38a5fd3249899a7963d1fd12b6e5 P 810ec5e7a7ca4e5d9eb11cfb4355d27e [term 1 FOLLOWER]: Refusing update from remote peer 986db6229294428dbfa6c776fd9d51e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:12:56.003607 11149 consensus_queue.cc:1035] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810ec5e7a7ca4e5d9eb11cfb4355d27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:56.003743 11151 consensus_queue.cc:1035] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fcdfc1094c43bbab49797d12032dc3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:12:56.006951 11156 mvcc.cc:205] Tried to move back new op lower bound from 7333225988106805248 to 7333225988060332032. Current Snapshot: MvccSnapshot[applied={T|T < 7333225988106805248}]
I20260925 12:12:56.008585 10668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:56.008684 10668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:12:56.009378 10668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e9b37d9e0dc438bbdb4e1705c46f280: Sending DeleteTablet for 3 replicas of tablet 99ca38a5fd3249899a7963d1fd12b6e5
I20260925 12:12:56.009727 10927 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 12:12:56.009752 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50164
I20260925 12:12:56.009745 11061 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ca38a5fd3249899a7963d1fd12b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46510
I20260925 12:12:56.009968 11164 tablet_replica.cc:330] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: stopping tablet replica
I20260925 12:12:56.010058 11164 raft_consensus.cc:2229] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:12:56.010179 11164 raft_consensus.cc:2258] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:56.010252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10730
I20260925 12:12:56.010339 11165 tablet_replica.cc:330] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: stopping tablet replica
I20260925 12:12:56.010877 11165 raft_consensus.cc:2229] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:12:56.010993 11165 raft_consensus.cc:2258] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:12:56.011548 11164 ts_tablet_manager.cc:1905] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:56.011551 11165 ts_tablet_manager.cc:1905] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:12:56.013486 11165 ts_tablet_manager.cc:1918] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:56.013528 11164 ts_tablet_manager.cc:1918] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:12:56.013615 11164 log.cc:1196] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/99ca38a5fd3249899a7963d1fd12b6e5
I20260925 12:12:56.013711 11165 log.cc:1196] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/99ca38a5fd3249899a7963d1fd12b6e5
I20260925 12:12:56.013886 11164 ts_tablet_manager.cc:1939] T 99ca38a5fd3249899a7963d1fd12b6e5 P 35fcdfc1094c43bbab49797d12032dc3: Deleting consensus metadata
I20260925 12:12:56.014088 11165 ts_tablet_manager.cc:1939] T 99ca38a5fd3249899a7963d1fd12b6e5 P 986db6229294428dbfa6c776fd9d51e9: Deleting consensus metadata
I20260925 12:12:56.014321 10653 catalog_manager.cc:4841] TS 35fcdfc1094c43bbab49797d12032dc3 (127.0.98.3:35453): tablet 99ca38a5fd3249899a7963d1fd12b6e5 (table test-table [id=146c979f16794103bf0a468abcaff510]) successfully deleted
I20260925 12:12:56.014493 10655 catalog_manager.cc:4841] TS 986db6229294428dbfa6c776fd9d51e9 (127.0.98.2:37727): tablet 99ca38a5fd3249899a7963d1fd12b6e5 (table test-table [id=146c979f16794103bf0a468abcaff510]) successfully deleted
W20260925 12:12:56.017050 10652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41357 (error 108)
W20260925 12:12:56.017608 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:12:56.017432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41357 (local address 127.0.0.1:50174)
0925 12:12:56.017508 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 12:12:56.017520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:12:56.017558 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41357: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 12:12:56.017715 10652 catalog_manager.cc:4584] TS 810ec5e7a7ca4e5d9eb11cfb4355d27e (127.0.98.1:41357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99ca38a5fd3249899a7963d1fd12b6e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41357: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 12:12:56.018031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10864
I20260925 12:12:56.022827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10998
I20260925 12:12:56.027271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 10636
2026-09-25T12:12:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1955 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:12:56 dist-test-slave-2hqq krb5kdc[11169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:12:56 dist-test-slave-2hqq krb5kdc[11169](info): set up 2 sockets
Sep 25 12:12:56 dist-test-slave-2hqq krb5kdc[11169](info): commencing operation
krb5kdc: starting...
W20260925 12:12:57.060832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:12:57 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:12:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 12:12:57.219306 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskODICQW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/postgres -l logfile start
2026-09-25 12:12:59.747 UTC [11199] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 12:12:59.747 UTC [11199] LOG: listening on IPv4 address "127.0.98.20", port 57959
2026-09-25 12:12:59.751 UTC [11199] LOG: listening on Unix socket "/tmp/.s.PGSQL.57959"
2026-09-25 12:12:59.785 UTC [11202] LOG: database system was shut down at 2026-09-25 12:12:58 UTC
2026-09-25 12:12:59.790 UTC [11199] LOG: database system is ready to accept connections
I20260925 12:13:00.800578 392 mini_postgres.cc:96] Postgres bound to 57959
2026-09-25 12:13:00.806 UTC [11210] FATAL: database "slave" does not exist
127.0.98.20:57959 - accepting connections
I20260925 12:13:00.806952 392 mini_ranger.cc:161] Starting Ranger...
I20260925 12:13:00.820839 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 12:13:01.306214 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 12:13:01.306299 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin
2026-09-25 12:13:01,575 [I] DB FLAVOR :POSTGRES
2026-09-25 12:13:01,576 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:13:01,576 [I] Checking connection..
2026-09-25 12:13:01,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 12:13:01,827 [I] Checking connection passed.
2026-09-25 12:13:01,827 [I] --------- Verifying version history table ---------
2026-09-25 12:13:01,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:02,073 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:13:02,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:02,318 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:13:02,318 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 12:13:02,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 12:13:02.561 UTC [11274] WARNING: there is no transaction in progress
2026-09-25 12:13:02,567 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 12:13:02,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:02,798 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 12:13:02,799 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:13:02,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 12:13:03,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 12:13:03,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2hqq.c.gcp-upstream.internal','N') ;"
2026-09-25 12:13:03,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:13:03,750 [I] Table x_portal_user does not exist in database ranger
2026-09-25 12:13:03,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:13:03,986 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 12:13:03,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:13:04,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:13:04,516 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 12:13:04,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 12:13:05.476 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.492 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.507 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.523 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.542 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.560 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.614 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.621 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.635 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.645 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.655 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.668 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.678 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.684 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.691 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:05.698 UTC [11382] WARNING: there is no transaction in progress
2026-09-25 12:13:06,305 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 12:13:06,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 12:13:06,539 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 12:13:06,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:13:06,780 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 12:13:06,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:13:07,021 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 12:13:07,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:13:07,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:13:07,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2hqq.c.gcp-upstream.internal';"
2026-09-25 12:13:07,739 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 12:13:07,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 12:13:07,996 [I] DB_PATCHES have already been applied
I20260925 12:13:08.002167 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskODICQW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 12:13:08.002244 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 12:13:08.005242 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37063
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:13:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 12:13:09.112046 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.305s user 0.003s sys 0.005s
Sep 25, 2026 12:13:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:13:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 12:13:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:13:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:13:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 12:13:30.706284 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 12:13:30 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338410, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 12:13:30.788825 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 12:13:30.797667 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 12:13:31.700776 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 12:13:31.700860 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms
I20260925 12:13:31.700881 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57959
2026-09-25 12:13:31,873 [I] DB FLAVOR :POSTGRES
2026-09-25 12:13:31,873 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:13:31,873 [I] Checking connection
2026-09-25 12:13:31,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 12:13:32,129 [I] connection success
2026-09-25 12:13:32,129 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 12:13:32,129 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 12:13:32,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 12:13:32,374 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 12:13:32,374 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:13:32,374 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 12:13:32,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 12:13:32,629 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 12:13:33.107214 392 mini_ranger_kms.cc:310] Created kms service
I20260925 12:13:33.293987 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 12:13:33.335522 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 12:13:33 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338410, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:13:33.715525 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 12:13:33.715788 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 12:13:33.715817 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 12:13:33.719405 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11562
I20260925 12:13:33.719527 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33213
Sep 25, 2026 12:13:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:13:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:13:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:13:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:13:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:13:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 12:13:36.003054 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.214s user 0.000s sys 0.154s
I20260925 12:13:36.003165 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 12:13:36.003244 392 mini_ranger_kms.cc:401] 127.0.98.20:33213/kms/v1/keys
Sep 25 12:13:36 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338416, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 12:13:36 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338416, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:13:36 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338416, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:13:39 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:13:39.104666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39707
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:13:39.202008 11613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:13:39.202194 11613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:13:39.202219 11613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:13:39.203320 11613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:13:39.203358 11613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:13:39.203373 11613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:13:39.203385 11613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:13:39.203421 11613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:13:39.204991 11613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39707
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:13:39.205210 11613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:13:39.205441 11613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:13:39.210884 11618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:39.210906 11621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:39.210975 11619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:39.211176 11613 server_base.cc:1034] running on GCE node
I20260925 12:13:39.211406 11613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:13:39.211704 11613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:13:39.212868 11613 hybrid_clock.cc:648] HybridClock initialized: now 1790338419212846 us; error 38 us; skew 500 ppm
Sep 25 12:13:39 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:13:39.221726 11613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:13:39.222216 11613 webserver.cc:460] Webserver started at http://127.0.98.62:42723/ using document root <none> and password file <none>
I20260925 12:13:39.222455 11613 fs_manager.cc:359] Metadata directory not provided
I20260925 12:13:39.222515 11613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:13:39.222671 11613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:13:39 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:13:39.282652 11613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a"
format_stamp: "Formatted at 2026-09-25 12:13:39 on dist-test-slave-2hqq"
server_key: "1252c005aecc60ae47407b2f95980fa0"
server_key_iv: "8a76c05dd3da2f45b83d64a4c75d94c7"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:39.283136 11613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a"
format_stamp: "Formatted at 2026-09-25 12:13:39 on dist-test-slave-2hqq"
server_key: "1252c005aecc60ae47407b2f95980fa0"
server_key_iv: "8a76c05dd3da2f45b83d64a4c75d94c7"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:39.284917 11613 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 12:13:39.301434 11628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:39.301611 11613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:13:39.301707 11613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a"
format_stamp: "Formatted at 2026-09-25 12:13:39 on dist-test-slave-2hqq"
server_key: "1252c005aecc60ae47407b2f95980fa0"
server_key_iv: "8a76c05dd3da2f45b83d64a4c75d94c7"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:39.301805 11613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:13:39.311808 11613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:13:39.314146 11613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:13:39.314432 11613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:13:39.320034 11613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39707
I20260925 12:13:39.320055 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39707 every 8 connection(s)
I20260925 12:13:39.320525 11613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:13:39.321165 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:13:39.323130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 11613
I20260925 12:13:39.323248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:13:39.323459 392 external_mini_cluster.cc:1428] Setting key 3878ea2f84e64a846d6a5105bfb2258a
I20260925 12:13:39.323541 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: Bootstrap starting.
I20260925 12:13:39.324252 11681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: Neither blocks nor log segments found. Creating new log.
I20260925 12:13:39.324571 11681 log.cc:824] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:13:39.325443 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: No bootstrap required, opened a new log
I20260925 12:13:39.327301 11681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } }
I20260925 12:13:39.327458 11681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:13:39.327487 11681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fe3b7863eb4d7bae79b5eef0d7bc6a, State: Initialized, Role: FOLLOWER
I20260925 12:13:39.327589 11681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } }
I20260925 12:13:39.327658 11681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:13:39.327764 11681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:13:39.327836 11681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:13:39.328471 11681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } }
I20260925 12:13:39.328579 11681 leader_election.cc:302] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13fe3b7863eb4d7bae79b5eef0d7bc6a; no voters:
I20260925 12:13:39.328711 11681 leader_election.cc:288] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:13:39.328853 11687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:13:39.328926 11681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:13:39.328992 11687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [term 1 LEADER]: Becoming Leader. State: Replica: 13fe3b7863eb4d7bae79b5eef0d7bc6a, State: Running, Role: LEADER
I20260925 12:13:39.329068 11687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } }
I20260925 12:13:39.329453 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } } }
I20260925 12:13:39.329557 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [sys.catalog]: This master's current role is: LEADER
I20260925 12:13:39.329696 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13fe3b7863eb4d7bae79b5eef0d7bc6a. Latest consensus state: current_term: 1 leader_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13fe3b7863eb4d7bae79b5eef0d7bc6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39707 } } }
I20260925 12:13:39.329772 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a [sys.catalog]: This master's current role is: LEADER
I20260925 12:13:39.330302 11681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 12:13:39 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:13:39.336104 11695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:13:39.337069 11695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:13:39.339625 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:39.324044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.0.98.62:39707)
0925 12:13:39.324203 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:39.324211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:39.324251 (+ 40us) server_negotiation.cc:407] Connection header received
0925 12:13:39.324877 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:39.324886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:39.324954 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:39.325107 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:13:39.325840 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:39.327630 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:39.329171 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:39.330067 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:39.330179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:39.334293 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:13:39.334338 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:13:39.334338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:13:39.334422 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:13:39.337525 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:39.337930 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:39.337933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:39.337934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:39.338042 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:39.338278 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:39.338279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:39.338279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:39.338593 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:13:39.339158 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:39.339229 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:13:39.339397 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260925 12:13:39.344553 11695 catalog_manager.cc:1347] Generated new cluster ID: 44bdab1ed0164ca8b66e4a94999a0079
I20260925 12:13:39.344605 11695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:13:39.368117 11695 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:13:39.368963 11695 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:13:39.384186 11695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: Generated new TSK 0
I20260925 12:13:39.384724 11695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:13:40.105556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:13:40.229307 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:13:40.229697 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:13:40.229785 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:13:40.231633 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:13:40.231751 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:13:40.231846 11726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:13:40.234107 11726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:13:40.234362 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:13:40.234648 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:13:40.240866 11734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:40.240866 11732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:40.241927 11726 server_base.cc:1034] running on GCE node
W20260925 12:13:40.242519 11731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:40.242802 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:13:40.243046 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:13:40.244222 11726 hybrid_clock.cc:648] HybridClock initialized: now 1790338420244200 us; error 32 us; skew 500 ppm
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:13:40.247938 11726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:13:40.248507 11726 webserver.cc:460] Webserver started at http://127.0.98.1:41277/ using document root <none> and password file <none>
I20260925 12:13:40.248803 11726 fs_manager.cc:359] Metadata directory not provided
I20260925 12:13:40.248863 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:13:40.249013 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:13:40.266572 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "c03c45340612465b8b195c2c26c72d61"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7b2f7d9b20f529da61d536d46054359b"
server_key_iv: "14c236884963400a65771b39628b6b23"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.267094 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c03c45340612465b8b195c2c26c72d61"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7b2f7d9b20f529da61d536d46054359b"
server_key_iv: "14c236884963400a65771b39628b6b23"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.268934 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 12:13:40.287495 11742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.287781 11726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 12:13:40.287910 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "c03c45340612465b8b195c2c26c72d61"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7b2f7d9b20f529da61d536d46054359b"
server_key_iv: "14c236884963400a65771b39628b6b23"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.288136 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:13:40.324369 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:13:40.326781 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:13:40.326985 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:13:40.327258 11726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:13:40.327618 11726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:13:40.327716 11726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.327811 11726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:13:40.327880 11726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.334739 11726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44599
I20260925 12:13:40.335204 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:13:40.338073 11856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44599 every 8 connection(s)
I20260925 12:13:40.343990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 11726
I20260925 12:13:40.344118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:13:40.344341 392 external_mini_cluster.cc:1428] Setting key 510557b10adf03f04bff1cfe4a7e1fb1
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:13:40.351917 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.339133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35023 (local address 127.0.98.62:39707)
0925 12:13:40.341117 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.341120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.341133 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:13:40.341376 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.341378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.341392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.341483 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:13:40.342437 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.343274 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.345060 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.345363 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.345437 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.347972 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:13:40.347974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:13:40.347974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:13:40.347981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:13:40.349783 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.350282 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.350283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.350284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.350348 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.351059 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.351060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.351060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.351205 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:13:40.351277 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.351722 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.351807 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1953,"thread_start_us":86,"threads_started":1}
I20260925 12:13:40.352084 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.339097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39707 (local address 127.0.98.1:35023)
0925 12:13:40.340616 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.340664 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.341247 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.341579 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.341584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.341844 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:13:40.342266 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.342273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.343391 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.343393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.344943 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.344946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.345509 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.345510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.345647 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.346253 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:13:40.346264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:13:40.347871 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:13:40.349874 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.349880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.349883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.350197 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.350411 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.350412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.350412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.350569 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.351369 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:13:40.351376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:13:40.351555 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.351685 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.351845 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"thread_start_us":97,"threads_started":1}
I20260925 12:13:40.352767 11857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39707
I20260925 12:13:40.352871 11857 heartbeater.cc:464] Registering TS with master...
I20260925 12:13:40.353047 11857 heartbeater.cc:511] Master 127.0.98.62:39707 requested a full tablet report, sending...
I20260925 12:13:40.353578 11645 ts_manager.cc:194] Registered new tserver with Master: c03c45340612465b8b195c2c26c72d61 (127.0.98.1:44599)
I20260925 12:13:40.354384 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35023
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:13:40.382929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 12:13:40.519769 11863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:13:40.520128 11863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:13:40.520182 11863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:13:40.521971 11863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:13:40.522099 11863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:13:40.522185 11863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:13:40.524441 11863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:13:40.524984 11863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:13:40.525337 11863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:13:40.531661 11873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:40.531629 11870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:40.531629 11871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:40.532124 11863 server_base.cc:1034] running on GCE node
I20260925 12:13:40.532397 11863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:13:40.532696 11863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:13:40.533869 11863 hybrid_clock.cc:648] HybridClock initialized: now 1790338420533850 us; error 37 us; skew 500 ppm
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:13:40.536937 11863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:13:40.537415 11863 webserver.cc:460] Webserver started at http://127.0.98.2:34425/ using document root <none> and password file <none>
I20260925 12:13:40.537703 11863 fs_manager.cc:359] Metadata directory not provided
I20260925 12:13:40.537760 11863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:13:40.537910 11863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:13:40.558295 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "d608617ec4fc4d7ebb4d581c4cd1b946"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7fdbad28bb72d69232173ecf3fa645f3"
server_key_iv: "a8655c91aa7e7896729f5fa9429599ab"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.558887 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d608617ec4fc4d7ebb4d581c4cd1b946"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7fdbad28bb72d69232173ecf3fa645f3"
server_key_iv: "a8655c91aa7e7896729f5fa9429599ab"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.560964 11863 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 12:13:40.585435 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.587463 11863 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.001s
I20260925 12:13:40.587595 11863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "d608617ec4fc4d7ebb4d581c4cd1b946"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "7fdbad28bb72d69232173ecf3fa645f3"
server_key_iv: "a8655c91aa7e7896729f5fa9429599ab"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.587730 11863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:13:40.617771 11863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:13:40.620281 11863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:13:40.620477 11863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:13:40.620807 11863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:13:40.621236 11863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:13:40.621335 11863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.621402 11863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:13:40.621456 11863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.628351 11863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46869
I20260925 12:13:40.632889 11994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46869 every 8 connection(s)
I20260925 12:13:40.633289 11863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:13:40.641651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 11863
I20260925 12:13:40.641753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:13:40.641947 392 external_mini_cluster.cc:1428] Setting key 55f187029158fcb8183d14e5158c6fd9
I20260925 12:13:40.645938 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.634466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33389 (local address 127.0.98.62:39707)
0925 12:13:40.634531 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.634534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.634782 (+ 248us) server_negotiation.cc:407] Connection header received
0925 12:13:40.635478 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.635479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.635491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.635522 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:13:40.636342 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.637170 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.638675 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.638989 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.639036 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.642321 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:13:40.642323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:13:40.642323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:13:40.642331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:13:40.644202 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.644745 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.644746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.644747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.644809 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.645084 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.645085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.645085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.645221 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:13:40.645264 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.645726 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.645834 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:13:40.646734 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.634469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39707 (local address 127.0.98.2:33389)
0925 12:13:40.634654 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.634697 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.635350 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.635719 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.635724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.635967 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:13:40.636230 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.636236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.637288 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.637289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.638551 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.638554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.639143 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.639144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.639335 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.639957 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:13:40.639967 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:13:40.642223 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:13:40.644303 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.644309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.644312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.644653 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.644878 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.644879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.644879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.645009 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.645311 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:13:40.645317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:13:40.645502 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.646302 (+ 800us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.646481 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260925 12:13:40.647589 11995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39707
I20260925 12:13:40.647699 11995 heartbeater.cc:464] Registering TS with master...
I20260925 12:13:40.647886 11995 heartbeater.cc:511] Master 127.0.98.62:39707 requested a full tablet report, sending...
I20260925 12:13:40.648366 11645 ts_manager.cc:194] Registered new tserver with Master: d608617ec4fc4d7ebb4d581c4cd1b946 (127.0.98.2:46869)
I20260925 12:13:40.648922 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33389
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:13:40.696388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:13:40.810668 12000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:13:40.810894 12000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:13:40.810931 12000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:13:40.812109 12000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:13:40.812196 12000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:13:40.812258 12000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:13:40.813802 12000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:13:40.814069 12000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:13:40.814289 12000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:13:40.820807 12006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:13:40.821223 12005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:40.822453 12000 server_base.cc:1034] running on GCE node
W20260925 12:13:40.831735 12008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:13:40.832197 12000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:13:40.832551 12000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:13:40.833722 12000 hybrid_clock.cc:648] HybridClock initialized: now 1790338420833698 us; error 41 us; skew 500 ppm
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:13:40.837083 12000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:13:40.837625 12000 webserver.cc:460] Webserver started at http://127.0.98.3:46785/ using document root <none> and password file <none>
I20260925 12:13:40.837893 12000 fs_manager.cc:359] Metadata directory not provided
I20260925 12:13:40.837944 12000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:13:40.838081 12000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:13:40.851766 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "7dc004b3a16f43e487f84454e7b84ff0"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "83aa90d34169d43ece35e12bad0e2dc9"
server_key_iv: "e0765da774c20ce4e095af0289315dbd"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.852542 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7dc004b3a16f43e487f84454e7b84ff0"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "83aa90d34169d43ece35e12bad0e2dc9"
server_key_iv: "e0765da774c20ce4e095af0289315dbd"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.854655 12000 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 12:13:40.872605 12015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.872875 12000 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260925 12:13:40.872960 12000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "7dc004b3a16f43e487f84454e7b84ff0"
format_stamp: "Formatted at 2026-09-25 12:13:40 on dist-test-slave-2hqq"
server_key: "83aa90d34169d43ece35e12bad0e2dc9"
server_key_iv: "e0765da774c20ce4e095af0289315dbd"
server_key_version: "kuduclusterkey@0"
I20260925 12:13:40.873065 12000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:13:40.886176 12000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:13:40.889587 12000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:13:40.889793 12000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:13:40.890096 12000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:13:40.890568 12000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:13:40.890655 12000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.890726 12000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:13:40.890781 12000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:13:40.905421 12000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33221
I20260925 12:13:40.906001 12000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:13:40.906966 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33221 every 8 connection(s)
I20260925 12:13:40.909564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 12000
I20260925 12:13:40.909663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:13:40.909870 392 external_mini_cluster.cc:1428] Setting key a980baf96b43fe14e41fcb01872407e3
Sep 25 12:13:40 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:13:40.918766 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.907889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59969 (local address 127.0.98.62:39707)
0925 12:13:40.907933 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.907935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.908357 (+ 422us) server_negotiation.cc:407] Connection header received
0925 12:13:40.909434 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.909435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.909447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.909477 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:13:40.910685 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.911440 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.912916 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.913188 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.913228 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.915664 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:13:40.915665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:13:40.915666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:13:40.915673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:13:40.917312 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.917767 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.917768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.917768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.917823 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.918083 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.918083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.918084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.918202 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:13:40.918241 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.918601 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.918677 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:13:40.920611 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.907639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39707 (local address 127.0.98.3:59969)
0925 12:13:40.908231 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.908275 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.909305 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.909656 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.909661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.909903 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:13:40.910376 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.910384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.911554 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.911555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.912820 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.912823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.913313 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.913314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.913441 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.914008 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:13:40.914019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:13:40.915560 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:13:40.917404 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.917410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.917413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.917682 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.917887 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:13:40.917887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:13:40.917888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:13:40.918003 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:13:40.918283 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:13:40.918289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:13:40.918455 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.919122 (+ 667us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.920280 (+ 1158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":74,"threads_started":1}
I20260925 12:13:40.921664 12129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39707
I20260925 12:13:40.921773 12129 heartbeater.cc:464] Registering TS with master...
I20260925 12:13:40.921993 12129 heartbeater.cc:511] Master 127.0.98.62:39707 requested a full tablet report, sending...
I20260925 12:13:40.922832 11645 ts_manager.cc:194] Registered new tserver with Master: 7dc004b3a16f43e487f84454e7b84ff0 (127.0.98.3:33221)
I20260925 12:13:40.923353 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59969
I20260925 12:13:40.932560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:13:40.948580 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.937469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.0.98.62:39707)
0925 12:13:40.937524 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.937526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.940897 (+ 3371us) server_negotiation.cc:407] Connection header received
0925 12:13:40.941005 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.941006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.941018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.941057 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:13:40.941907 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.942649 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.943718 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.943995 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.944035 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.945629 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:13:40.945630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:13:40.945631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:13:40.945639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:13:40.947305 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.947670 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.947671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.947672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.947736 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:13:40.947938 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:13:40.947938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:13:40.947939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:13:40.948083 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:13:40.948155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.948348 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.948421 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:13:40.950872 11645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:13:40.953287 11645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:13:40.966617 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.956597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33221 (local address 127.0.0.1:35742)
0925 12:13:40.957895 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.957904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.957946 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.958206 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.958209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.958211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:13:40.958274 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.958276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.959751 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.959752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.961028 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.961031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.966298 (+ 5267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.966299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.966385 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.966398 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.966465 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.966517 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"thread_start_us":61,"threads_started":1}
I20260925 12:13:40.967077 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.956989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.0.98.3:33221)
0925 12:13:40.957786 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.957791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.957994 (+ 203us) server_negotiation.cc:407] Connection header received
0925 12:13:40.958038 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.958040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.958057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.958119 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:13:40.958343 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.959671 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.961736 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.966140 (+ 4404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.966198 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.966267 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.966950 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.966995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":68,"threads_started":1}
I20260925 12:13:40.968130 12063 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 (DEFAULT_TABLE table=test-table [id=efda4e6fc62d4671a33c6002bdcb432e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:13:40.968480 12063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7a2e96f32468a98cdea939efea2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:13:40.973281 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.956357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46869 (local address 127.0.0.1:48828)
0925 12:13:40.957552 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.957594 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.957652 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.969053 (+ 11401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.969054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.969057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:13:40.969120 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.969123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.970417 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.970418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.971920 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.971922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.973021 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.973022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.973097 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.973101 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.973149 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.973196 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":63,"threads_started":1}
I20260925 12:13:40.973703 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.956449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.0.98.2:46869)
0925 12:13:40.968718 (+ 12269us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.968724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.968736 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:13:40.968780 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.968782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.968799 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.968906 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:13:40.969216 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.970305 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.972059 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.972827 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.972906 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.972977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.973570 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.973621 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12220,"thread_start_us":122,"threads_started":1}
I20260925 12:13:40.973766 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.956727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44599 (local address 127.0.0.1:58226)
0925 12:13:40.961145 (+ 4418us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:40.961155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:40.961193 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:40.962501 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:40.962502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:40.962504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:13:40.962560 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.962562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.963702 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.963703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.965072 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:40.965074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.973430 (+ 8356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.973430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:40.973502 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.973506 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:13:40.973648 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 12:13:40.973688 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4401,"thread_start_us":55,"threads_started":1}
I20260925 12:13:40.974035 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:40.957129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.0.98.1:44599)
0925 12:13:40.957851 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:40.957855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:40.961248 (+ 3393us) server_negotiation.cc:407] Connection header received
0925 12:13:40.961285 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:40.961288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:40.961304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:40.961641 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:13:40.962646 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.963610 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.965166 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:40.965895 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:40.965962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:40.966020 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:40.973902 (+ 7882us) server_negotiation.cc:299] Negotiation successful
0925 12:13:40.973959 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":45,"threads_started":1}
I20260925 12:13:40.974591 11929 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 (DEFAULT_TABLE table=test-table [id=efda4e6fc62d4671a33c6002bdcb432e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:13:40.974743 11791 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 (DEFAULT_TABLE table=test-table [id=efda4e6fc62d4671a33c6002bdcb432e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:13:40.974929 11929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7a2e96f32468a98cdea939efea2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:13:40.975032 11791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7a2e96f32468a98cdea939efea2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:13:40.978751 12150 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Bootstrap starting.
I20260925 12:13:40.978945 12148 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Bootstrap starting.
I20260925 12:13:40.979674 12150 tablet_bootstrap.cc:654] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Neither blocks nor log segments found. Creating new log.
I20260925 12:13:40.980681 12148 tablet_bootstrap.cc:654] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Neither blocks nor log segments found. Creating new log.
I20260925 12:13:40.981040 12150 log.cc:824] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Log is configured to *not* fsync() on all Append() calls
I20260925 12:13:40.981590 12148 log.cc:824] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Log is configured to *not* fsync() on all Append() calls
I20260925 12:13:40.982352 12150 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: No bootstrap required, opened a new log
I20260925 12:13:40.982478 12150 ts_tablet_manager.cc:1397] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 12:13:40.984695 12149 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Bootstrap starting.
I20260925 12:13:40.985348 12150 raft_consensus.cc:348] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:40.985623 12150 raft_consensus.cc:374] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:13:40.985702 12150 raft_consensus.cc:729] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03c45340612465b8b195c2c26c72d61, State: Initialized, Role: FOLLOWER
I20260925 12:13:40.985811 12149 tablet_bootstrap.cc:654] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Neither blocks nor log segments found. Creating new log.
I20260925 12:13:40.985886 12150 consensus_queue.cc:260] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:40.986214 12150 ts_tablet_manager.cc:1428] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 12:13:40.986362 11857 heartbeater.cc:503] Master 127.0.98.62:39707 was elected leader, sending a full tablet report...
I20260925 12:13:40.988058 12148 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: No bootstrap required, opened a new log
I20260925 12:13:40.988183 12148 ts_tablet_manager.cc:1397] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 12:13:40.988838 12149 log.cc:824] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Log is configured to *not* fsync() on all Append() calls
I20260925 12:13:40.990934 12148 raft_consensus.cc:348] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:40.991173 12148 raft_consensus.cc:374] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:13:40.991250 12148 raft_consensus.cc:729] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc004b3a16f43e487f84454e7b84ff0, State: Initialized, Role: FOLLOWER
I20260925 12:13:40.991426 12148 consensus_queue.cc:260] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:40.991801 12148 ts_tablet_manager.cc:1428] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 12:13:40.991990 12129 heartbeater.cc:503] Master 127.0.98.62:39707 was elected leader, sending a full tablet report...
I20260925 12:13:41.001365 12149 tablet_bootstrap.cc:492] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: No bootstrap required, opened a new log
I20260925 12:13:41.001482 12149 ts_tablet_manager.cc:1397] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260925 12:13:41.003468 12149 raft_consensus.cc:348] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:41.003662 12149 raft_consensus.cc:374] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:13:41.003692 12149 raft_consensus.cc:729] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d608617ec4fc4d7ebb4d581c4cd1b946, State: Initialized, Role: FOLLOWER
I20260925 12:13:41.003804 12149 consensus_queue.cc:260] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:41.004068 12149 ts_tablet_manager.cc:1428] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:13:41.004175 11995 heartbeater.cc:503] Master 127.0.98.62:39707 was elected leader, sending a full tablet report...
Sep 25 12:13:41 dist-test-slave-2hqq krb5kdc[11169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 12:13:41.089342 11858 tablet.cc:2369] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:13:41.105589 12153 raft_consensus.cc:482] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:13:41.105746 12153 raft_consensus.cc:504] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:41.106092 12153 leader_election.cc:288] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d608617ec4fc4d7ebb4d581c4cd1b946 (127.0.98.2:46869), 7dc004b3a16f43e487f84454e7b84ff0 (127.0.98.3:33221)
I20260925 12:13:41.112938 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.106924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40987 (local address 127.0.98.2:46869)
0925 12:13:41.106979 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:41.106982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:41.107179 (+ 197us) server_negotiation.cc:407] Connection header received
0925 12:13:41.107282 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:41.107284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:41.107297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:41.107325 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:13:41.107536 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.108236 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.111938 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.112512 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.112560 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:41.112572 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:41.112823 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 12:13:41.112865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:13:41.113417 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.106387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46869 (local address 127.0.98.1:40987)
0925 12:13:41.107104 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:41.107115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:41.107208 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:41.107392 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:41.107393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:41.107394 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:13:41.107473 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:41.107475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.108357 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.108358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:41.109514 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:41.109518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.112628 (+ 3110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.112628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:41.112697 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:41.112701 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:13:41.113296 (+ 595us) client_negotiation.cc:240] Negotiation successful
0925 12:13:41.113339 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":106,"threads_started":1}
I20260925 12:13:41.114737 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.106979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60963 (local address 127.0.98.3:33221)
0925 12:13:41.107020 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:41.107023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:41.109680 (+ 2657us) server_negotiation.cc:407] Connection header received
0925 12:13:41.109768 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:41.109769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:41.109778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:41.109814 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:13:41.110004 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.110628 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.113660 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.114162 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.114210 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:41.114219 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:41.114623 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 12:13:41.114664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:13:41.114696 11949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7a2e96f32468a98cdea939efea2b5" candidate_uuid: "c03c45340612465b8b195c2c26c72d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" is_pre_election: true
I20260925 12:13:41.114874 11949 raft_consensus.cc:2454] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03c45340612465b8b195c2c26c72d61 in term 0.
I20260925 12:13:41.115106 11746 leader_election.cc:302] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c03c45340612465b8b195c2c26c72d61, d608617ec4fc4d7ebb4d581c4cd1b946; no voters:
I20260925 12:13:41.115118 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.106603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33221 (local address 127.0.98.1:60963)
0925 12:13:41.109627 (+ 3024us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:41.109638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:13:41.109704 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:13:41.109873 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:13:41.109874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:13:41.109875 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:13:41.109939 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:41.109940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.110752 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.110753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:41.111832 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:13:41.111835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.114277 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.114278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:13:41.114348 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:41.114351 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:13:41.114988 (+ 637us) client_negotiation.cc:240] Negotiation successful
0925 12:13:41.115033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3008,"thread_start_us":71,"threads_started":1}
I20260925 12:13:41.115295 12153 raft_consensus.cc:2781] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:13:41.115348 12153 raft_consensus.cc:482] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:13:41.115370 12153 raft_consensus.cc:3037] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:13:41.116267 12153 raft_consensus.cc:504] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:41.116393 12153 leader_election.cc:288] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [CANDIDATE]: Term 1 election: Requested vote from peers d608617ec4fc4d7ebb4d581c4cd1b946 (127.0.98.2:46869), 7dc004b3a16f43e487f84454e7b84ff0 (127.0.98.3:33221)
I20260925 12:13:41.116750 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7a2e96f32468a98cdea939efea2b5" candidate_uuid: "c03c45340612465b8b195c2c26c72d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dc004b3a16f43e487f84454e7b84ff0" is_pre_election: true
I20260925 12:13:41.116881 12067 raft_consensus.cc:2454] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03c45340612465b8b195c2c26c72d61 in term 0.
I20260925 12:13:41.117398 11949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7a2e96f32468a98cdea939efea2b5" candidate_uuid: "c03c45340612465b8b195c2c26c72d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946"
I20260925 12:13:41.117475 11949 raft_consensus.cc:3037] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:13:41.118333 11949 raft_consensus.cc:2454] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03c45340612465b8b195c2c26c72d61 in term 1.
I20260925 12:13:41.118553 11746 leader_election.cc:302] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c03c45340612465b8b195c2c26c72d61, d608617ec4fc4d7ebb4d581c4cd1b946; no voters:
I20260925 12:13:41.118700 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7a2e96f32468a98cdea939efea2b5" candidate_uuid: "c03c45340612465b8b195c2c26c72d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dc004b3a16f43e487f84454e7b84ff0"
I20260925 12:13:41.118773 12067 raft_consensus.cc:3037] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:13:41.119591 12067 raft_consensus.cc:2454] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03c45340612465b8b195c2c26c72d61 in term 1.
I20260925 12:13:41.119838 12153 raft_consensus.cc:2781] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:13:41.119987 12153 raft_consensus.cc:686] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [term 1 LEADER]: Becoming Leader. State: Replica: c03c45340612465b8b195c2c26c72d61, State: Running, Role: LEADER
I20260925 12:13:41.120076 12153 consensus_queue.cc:237] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } }
I20260925 12:13:41.120903 11645 catalog_manager.cc:5495] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to c03c45340612465b8b195c2c26c72d61 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c03c45340612465b8b195c2c26c72d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03c45340612465b8b195c2c26c72d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } health_report { overall_health: HEALTHY } } }
W20260925 12:13:41.134364 11996 tablet.cc:2369] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:13:41.146435 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.143876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.0.98.1:44599)
0925 12:13:41.144012 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 12:13:41.144014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:13:41.144020 (+ 6us) server_negotiation.cc:407] Connection header received
0925 12:13:41.144052 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:13:41.144053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:13:41.144072 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:13:41.144151 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:13:41.144295 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.144852 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.145611 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:13:41.145820 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:13:41.145904 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:13:41.146081 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:13:41.146194 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:13:41.146291 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 12:13:41.146309 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:13:41.146331 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 12:13:41.148917 11949 raft_consensus.cc:1261] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 1 FOLLOWER]: Refusing update from remote peer c03c45340612465b8b195c2c26c72d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:13:41.148917 12067 raft_consensus.cc:1261] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 1 FOLLOWER]: Refusing update from remote peer c03c45340612465b8b195c2c26c72d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:13:41.149259 12153 consensus_queue.cc:1035] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d608617ec4fc4d7ebb4d581c4cd1b946" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:13:41.149385 12159 consensus_queue.cc:1035] T 63b7a2e96f32468a98cdea939efea2b5 P c03c45340612465b8b195c2c26c72d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc004b3a16f43e487f84454e7b84ff0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:13:41.153187 12165 mvcc.cc:205] Tried to move back new op lower bound from 7333226173022580736 to 7333226172908490752. Current Snapshot: MvccSnapshot[applied={T|T < 7333226173022580736}]
I20260925 12:13:41.153405 12162 mvcc.cc:205] Tried to move back new op lower bound from 7333226173022580736 to 7333226172908490752. Current Snapshot: MvccSnapshot[applied={T|T < 7333226173022580736}]
I20260925 12:13:41.153606 12163 mvcc.cc:205] Tried to move back new op lower bound from 7333226173022580736 to 7333226172908490752. Current Snapshot: MvccSnapshot[applied={T|T < 7333226173022580736}]
I20260925 12:13:41.158987 11644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:13:41.159096 11644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:13:41.159925 11644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13fe3b7863eb4d7bae79b5eef0d7bc6a: Sending DeleteTablet for 3 replicas of tablet 63b7a2e96f32468a98cdea939efea2b5
I20260925 12:13:41.160475 11929 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:13:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48828
I20260925 12:13:41.160722 12063 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:13:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35742
I20260925 12:13:41.160902 11791 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7a2e96f32468a98cdea939efea2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:13:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58226
I20260925 12:13:41.161588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 11726
I20260925 12:13:41.163044 12172 tablet_replica.cc:330] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: stopping tablet replica
I20260925 12:13:41.163060 12173 tablet_replica.cc:330] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: stopping tablet replica
I20260925 12:13:41.163280 12173 raft_consensus.cc:2229] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:13:41.163431 12173 raft_consensus.cc:2258] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:13:41.163569 12172 raft_consensus.cc:2229] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:13:41.163689 12172 raft_consensus.cc:2258] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:13:41.164230 12172 ts_tablet_manager.cc:1905] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:13:41.164883 12173 ts_tablet_manager.cc:1905] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:13:41.166087 12172 ts_tablet_manager.cc:1918] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:13:41.166174 12172 log.cc:1196] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/63b7a2e96f32468a98cdea939efea2b5
I20260925 12:13:41.166436 12172 ts_tablet_manager.cc:1939] T 63b7a2e96f32468a98cdea939efea2b5 P d608617ec4fc4d7ebb4d581c4cd1b946: Deleting consensus metadata
I20260925 12:13:41.166618 12173 ts_tablet_manager.cc:1918] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:13:41.166698 12173 log.cc:1196] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/63b7a2e96f32468a98cdea939efea2b5
I20260925 12:13:41.166890 11631 catalog_manager.cc:4841] TS d608617ec4fc4d7ebb4d581c4cd1b946 (127.0.98.2:46869): tablet 63b7a2e96f32468a98cdea939efea2b5 (table test-table [id=efda4e6fc62d4671a33c6002bdcb432e]) successfully deleted
I20260925 12:13:41.167704 12173 ts_tablet_manager.cc:1939] T 63b7a2e96f32468a98cdea939efea2b5 P 7dc004b3a16f43e487f84454e7b84ff0: Deleting consensus metadata
I20260925 12:13:41.168169 11629 catalog_manager.cc:4841] TS 7dc004b3a16f43e487f84454e7b84ff0 (127.0.98.3:33221): tablet 63b7a2e96f32468a98cdea939efea2b5 (table test-table [id=efda4e6fc62d4671a33c6002bdcb432e]) successfully deleted
W20260925 12:13:41.171411 11629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44599 (error 108)
I20260925 12:13:41.171849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 11863
I20260925 12:13:41.172796 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:13:41.172649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44599 (local address 127.0.0.1:58250)
0925 12:13:41.172696 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:13:41.172747 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 12:13:41.173019 11629 catalog_manager.cc:4584] TS c03c45340612465b8b195c2c26c72d61 (127.0.98.1:44599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63b7a2e96f32468a98cdea939efea2b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44599: connect: Connection refused (error 111)
I20260925 12:13:41.176764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 12000
I20260925 12:13:41.181988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 11613
I20260925 12:13:41.186506 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 12:13:41.530905 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 12:13:41.531026 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 12:13:41.875319 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 12:13:41.875 UTC [11199] LOG: received smart shutdown request
2026-09-25 12:13:41.879 UTC [11199] LOG: background worker "logical replication launcher" (PID 11208) exited with exit code 1
2026-09-25 12:13:41.879 UTC [11203] LOG: shutting down
2026-09-25 12:13:42.053 UTC [11199] LOG: database system is shut down
2026-09-25T12:13:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46130 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:13:42 dist-test-slave-2hqq krb5kdc[12184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:13:42 dist-test-slave-2hqq krb5kdc[12184](info): set up 2 sockets
Sep 25 12:13:42 dist-test-slave-2hqq krb5kdc[12184](info): commencing operation
krb5kdc: starting...
W20260925 12:13:43.180748 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:13:43 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:13:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 12:13:43.341348 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskODICQW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/postgres -l logfile start
2026-09-25 12:13:45.850 UTC [12214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 12:13:45.850 UTC [12214] LOG: listening on IPv4 address "127.0.98.20", port 42855
2026-09-25 12:13:45.854 UTC [12214] LOG: listening on Unix socket "/tmp/.s.PGSQL.42855"
2026-09-25 12:13:45.889 UTC [12217] LOG: database system was shut down at 2026-09-25 12:13:44 UTC
2026-09-25 12:13:45.893 UTC [12214] LOG: database system is ready to accept connections
I20260925 12:13:46.844476 392 mini_postgres.cc:96] Postgres bound to 42855
2026-09-25 12:13:46.850 UTC [12225] FATAL: database "slave" does not exist
127.0.98.20:42855 - accepting connections
I20260925 12:13:46.850883 392 mini_ranger.cc:161] Starting Ranger...
I20260925 12:13:46.865796 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 12:13:47.368353 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 12:13:47.368440 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin
2026-09-25 12:13:47,647 [I] DB FLAVOR :POSTGRES
2026-09-25 12:13:47,647 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:13:47,647 [I] Checking connection..
2026-09-25 12:13:47,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 12:13:47,876 [I] Checking connection passed.
2026-09-25 12:13:47,876 [I] --------- Verifying version history table ---------
2026-09-25 12:13:47,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:48,131 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:13:48,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:48,374 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 12:13:48,374 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 12:13:48,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 12:13:48.611 UTC [12289] WARNING: there is no transaction in progress
2026-09-25 12:13:48,617 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 12:13:48,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 12:13:48,859 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 12:13:48,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:13:48,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 12:13:49,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 12:13:49,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2hqq.c.gcp-upstream.internal','N') ;"
2026-09-25 12:13:49,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:13:49,877 [I] Table x_portal_user does not exist in database ranger
2026-09-25 12:13:49,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:13:50,124 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 12:13:50,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:13:50,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:13:50,573 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 12:13:50,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 12:13:51.564 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.581 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.597 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.612 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.628 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.646 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.703 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.710 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.725 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.737 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.748 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.762 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.771 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.778 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.785 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:51.791 UTC [12397] WARNING: there is no transaction in progress
2026-09-25 12:13:52,463 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 12:13:52,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 12:13:52,741 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 12:13:52,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 12:13:52,978 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 12:13:52,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 12:13:53,226 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 12:13:53,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 12:13:53,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 12:13:53,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2hqq.c.gcp-upstream.internal';"
2026-09-25 12:13:53,971 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 12:13:53,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 12:13:54,208 [I] DB_PATCHES have already been applied
I20260925 12:13:54.214181 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskODICQW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 12:13:54.214247 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 12:13:54.217602 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45255
Sep 25, 2026 12:13:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:13:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:13:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 12:13:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 12:13:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 12:13:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 12:13:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 12:13:55.314008 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.463s user 0.001s sys 0.008s
Sep 25, 2026 12:14:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 12:14:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 12:14:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 12:14:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 12:14:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 12:14:17.348407 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 12:14:17 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338457, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 12:14:17.443496 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 12:14:17.453652 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 12:14:18.380368 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 12:14:18.380453 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms
I20260925 12:14:18.380471 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42855
2026-09-25 12:14:18,578 [I] DB FLAVOR :POSTGRES
2026-09-25 12:14:18,578 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 12:14:18,578 [I] Checking connection
2026-09-25 12:14:18,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 12:14:18,842 [I] connection success
2026-09-25 12:14:18,842 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 12:14:18,842 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 12:14:18,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 12:14:19,117 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 12:14:19,117 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 12:14:19,117 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 12:14:19,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 12:14:19,393 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 12:14:19.888046 392 mini_ranger_kms.cc:310] Created kms service
I20260925 12:14:20.065805 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 12:14:20.114270 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 12:14:20 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338457, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:20.542716 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 12:14:20.542838 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskODICQW/build/release/bin/postgresql.jar:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskODICQW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 12:14:20.542856 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 12:14:20.546610 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12577
I20260925 12:14:20.546753 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53223
Sep 25, 2026 12:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 12:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 12:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 12:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 12:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 12:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 12:14:22.784291 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.341s user 0.012s sys 0.168s
I20260925 12:14:22.784410 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 12:14:22.784461 392 mini_ranger_kms.cc:401] 127.0.98.20:53223/kms/v1/keys
Sep 25 12:14:22 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 12:14:23 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:14:23 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338462, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 12:14:25 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:14:25.766999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:25.860702 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:25.860935 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:25.860975 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:25.862183 12628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:25.862248 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:25.862273 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:25.862313 12628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:25.862340 12628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:25.863956 12628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53223/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:25.864230 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:25.864463 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:25.869709 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:25.869704 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:25.869871 12628 server_base.cc:1034] running on GCE node
W20260925 12:14:25.869884 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:25.870213 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:25.870548 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:25.871729 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790338465871711 us; error 46 us; skew 500 ppm
Sep 25 12:14:25 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:25.879882 12628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:25.880613 12628 webserver.cc:460] Webserver started at http://127.0.98.62:35517/ using document root <none> and password file <none>
I20260925 12:14:25.880887 12628 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:25.880928 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:25.881083 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:14:25 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:25.927470 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "eba9391c91e042018566bdc0a367d246"
format_stamp: "Formatted at 2026-09-25 12:14:25 on dist-test-slave-2hqq"
server_key: "30d777883472edf1f6ab58aaef8c05e6"
server_key_iv: "cb57184578ef534552b4b40312d3dcbe"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:25.927896 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "eba9391c91e042018566bdc0a367d246"
format_stamp: "Formatted at 2026-09-25 12:14:25 on dist-test-slave-2hqq"
server_key: "30d777883472edf1f6ab58aaef8c05e6"
server_key_iv: "cb57184578ef534552b4b40312d3dcbe"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:25.929484 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:14:25.942086 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:25.942332 12628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:25.942433 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "eba9391c91e042018566bdc0a367d246"
format_stamp: "Formatted at 2026-09-25 12:14:25 on dist-test-slave-2hqq"
server_key: "30d777883472edf1f6ab58aaef8c05e6"
server_key_iv: "cb57184578ef534552b4b40312d3dcbe"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:25.942536 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:25.970611 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:25.973388 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:25.973595 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:25.979081 12628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260925 12:14:25.979161 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260925 12:14:25.979563 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:25.980345 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:25.982818 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Bootstrap starting.
I20260925 12:14:25.983570 12696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:25.983570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 12628
I20260925 12:14:25.983670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:14:25.983915 392 external_mini_cluster.cc:1428] Setting key 1afd5da21e58c7dbdc817280c5a62fcc
I20260925 12:14:25.983959 12696 log.cc:824] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:25.985081 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: No bootstrap required, opened a new log
I20260925 12:14:25.987038 12696 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:25.987224 12696 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:25.987275 12696 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba9391c91e042018566bdc0a367d246, State: Initialized, Role: FOLLOWER
I20260925 12:14:25.987411 12696 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:25.987488 12696 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:25.987512 12696 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:25.987542 12696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:25.988070 12696 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:25.988164 12696 leader_election.cc:302] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba9391c91e042018566bdc0a367d246; no voters:
I20260925 12:14:25.988327 12696 leader_election.cc:288] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:14:25.988562 12702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:25.988718 12702 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 LEADER]: Becoming Leader. State: Replica: eba9391c91e042018566bdc0a367d246, State: Running, Role: LEADER
I20260925 12:14:25.988781 12696 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:25.988801 12702 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:25.989138 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba9391c91e042018566bdc0a367d246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260925 12:14:25.989219 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:25.989331 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba9391c91e042018566bdc0a367d246. Latest consensus state: current_term: 1 leader_uuid: "eba9391c91e042018566bdc0a367d246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260925 12:14:25.989365 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:25.989465 12706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:25.990234 12706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:25.991161 12696 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 12:14:25 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:25.996420 12706 catalog_manager.cc:1347] Generated new cluster ID: e89f02575e004b0e8ba75607071a49e3
I20260925 12:14:25.996589 12706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:25.998739 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:25.984479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.0.98.62:42643)
0925 12:14:25.984707 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:25.984718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:25.984754 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:14:25.985329 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:25.985338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:25.985403 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:25.985555 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:25.986403 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:25.988259 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:25.989675 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:25.990386 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:25.990511 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:25.993515 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:25.993569 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:25.993569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:25.993640 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:25.997018 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:25.997375 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:25.997379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:25.997379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:25.997486 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:25.997710 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:25.997711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:25.997711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:25.998055 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:25.998215 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:25.998321 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 12:14:25.998471 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":122,"threads_started":1}
I20260925 12:14:26.012888 12706 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:14:26.013653 12706 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:26.016184 12706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Generated new TSK 0
I20260925 12:14:26.016408 12706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:14:26.728046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:26.833954 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:26.834193 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:26.834235 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:26.835434 12741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:26.835505 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:26.835548 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:26.837088 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:26.837340 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:26.837575 12741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:26.842650 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:26.842995 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:26.843094 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:26.843266 12741 server_base.cc:1034] running on GCE node
I20260925 12:14:26.843492 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:26.843762 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:26.844995 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790338466844997 us; error 56 us; skew 500 ppm
Sep 25 12:14:26 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:26.847723 12741 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:26.848268 12741 webserver.cc:460] Webserver started at http://127.0.98.1:33037/ using document root <none> and password file <none>
I20260925 12:14:26.848557 12741 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:26.848626 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:26.848769 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:14:26 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:26.861753 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "3618a79426e242d19681d3d70ad7a5c8"
format_stamp: "Formatted at 2026-09-25 12:14:26 on dist-test-slave-2hqq"
server_key: "d030c82fd489016b3043211d3a6f64b6"
server_key_iv: "2e29032f504a628753f4239285e91239"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:26.862200 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3618a79426e242d19681d3d70ad7a5c8"
format_stamp: "Formatted at 2026-09-25 12:14:26 on dist-test-slave-2hqq"
server_key: "d030c82fd489016b3043211d3a6f64b6"
server_key_iv: "2e29032f504a628753f4239285e91239"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:26.864661 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 12:14:26.889536 12758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:26.889842 12741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:26.889968 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "3618a79426e242d19681d3d70ad7a5c8"
format_stamp: "Formatted at 2026-09-25 12:14:26 on dist-test-slave-2hqq"
server_key: "d030c82fd489016b3043211d3a6f64b6"
server_key_iv: "2e29032f504a628753f4239285e91239"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:26.890086 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:26.921783 12741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:26.924396 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:26.924690 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:26.924989 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:26.925376 12741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:26.925467 12741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:26.925565 12741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:26.925652 12741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:26.932544 12741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35367
I20260925 12:14:26.933090 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:14:26.933666 12871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35367 every 8 connection(s)
I20260925 12:14:26.934423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 12741
I20260925 12:14:26.934515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:14:26.934712 392 external_mini_cluster.cc:1428] Setting key fa1ae205fea32b411a690b3710454e9c
Sep 25 12:14:26 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:26.949299 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:26.935712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37293 (local address 127.0.98.62:42643)
0925 12:14:26.937921 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:26.937923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:26.937931 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:14:26.937954 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:26.937954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:26.937968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:26.938036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:26.938876 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:26.939730 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:26.941356 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:26.941683 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:26.941747 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:26.945636 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:26.945638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:26.945638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:26.945647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:26.947479 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:26.947958 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:26.947959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:26.947960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:26.948030 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:26.948302 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:26.948303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:26.948303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:26.948448 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:26.948534 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:26.949109 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 12:14:26.949190 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":72,"threads_started":1}
I20260925 12:14:26.949921 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:26.935442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:37293)
0925 12:14:26.937045 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:26.937094 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:26.937743 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:26.938167 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:26.938172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:26.938434 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:26.938768 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:26.938775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:26.939935 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:26.939936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:26.941259 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:26.941262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:26.942963 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:26.942965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:26.943117 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:26.943746 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:26.943757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:26.945513 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:26.947574 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:26.947582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:26.947585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:26.947873 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:26.948098 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:26.948099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:26.948099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:26.948233 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:26.948560 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:26.948567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:26.948752 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 12:14:26.948853 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:14:26.949676 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":78,"threads_started":1}
I20260925 12:14:26.950619 12872 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:26.950713 12872 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:26.950896 12872 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:26.951462 12660 ts_manager.cc:194] Registered new tserver with Master: 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367)
I20260925 12:14:26.952286 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37293
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:14:26.983721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 12:14:27.074047 12880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:27.074339 12880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:27.074380 12880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:27.076176 12880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:27.076301 12880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:27.076378 12880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:27.078685 12880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:27.079097 12880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:27.079432 12880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:27.085414 12888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.085326 12885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.085326 12886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:27.086050 12880 server_base.cc:1034] running on GCE node
I20260925 12:14:27.086305 12880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:27.086588 12880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:27.087759 12880 hybrid_clock.cc:648] HybridClock initialized: now 1790338467087738 us; error 33 us; skew 500 ppm
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:27.091317 12880 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:27.091955 12880 webserver.cc:460] Webserver started at http://127.0.98.2:33253/ using document root <none> and password file <none>
I20260925 12:14:27.092195 12880 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:27.092276 12880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:27.092422 12880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:27.105333 12880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "0e8395a0e6da48ac4e618ec8e1d35e51"
server_key_iv: "deb4683551f953f4435ad69882fb5337"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.105790 12880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "0e8395a0e6da48ac4e618ec8e1d35e51"
server_key_iv: "deb4683551f953f4435ad69882fb5337"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.108865 12880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 12:14:27.121621 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.121898 12880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:27.122014 12880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "0e8395a0e6da48ac4e618ec8e1d35e51"
server_key_iv: "deb4683551f953f4435ad69882fb5337"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.122120 12880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:27.144438 12880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:27.147060 12880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:27.147239 12880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:27.147541 12880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:27.147930 12880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:27.147981 12880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.148013 12880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:27.148047 12880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.155587 12880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40557
I20260925 12:14:27.155768 13009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40557 every 8 connection(s)
I20260925 12:14:27.156015 12880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:14:27.161682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 12880
I20260925 12:14:27.161772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:14:27.162405 392 external_mini_cluster.cc:1428] Setting key 24a9bf8accf06286644ba4e2cbf9747b
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:27.167637 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.156787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:36483)
0925 12:14:27.156982 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:27.157032 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:27.157721 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:27.158112 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:27.158117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:27.158363 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:27.158663 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.158670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.159701 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.159702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.160993 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.160995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.161562 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.161563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.161708 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.162312 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:27.162322 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:27.163952 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:27.165965 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.165972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.165974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.166280 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.166509 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.166509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.166510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.166651 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.166950 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:27.166957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:27.167131 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 12:14:27.167240 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 12:14:27.167402 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260925 12:14:27.168099 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.156825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36483 (local address 127.0.98.62:42643)
0925 12:14:27.156908 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:27.156910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:27.157129 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:14:27.157863 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:27.157864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:27.157878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:27.157910 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:27.158783 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.159581 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.161096 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.161403 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.161455 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.164063 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:27.164065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:27.164066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:27.164074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:27.165869 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.166366 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.166367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.166367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.166437 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.166727 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.166728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.166728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.166866 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:27.166925 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:27.167901 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 12:14:27.167992 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 12:14:27.168444 13010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:27.168565 13010 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:27.168752 13010 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:27.169178 12660 ts_manager.cc:194] Registered new tserver with Master: bf1f8ac5e1f341f7b1f81bddd98784c4 (127.0.98.2:40557)
I20260925 12:14:27.169656 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36483
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:14:27.196341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:27.318694 13015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:27.319056 13015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:27.319124 13015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:27.321034 13015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:27.321103 13015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:27.321148 13015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:27.323473 13015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:27.323736 13015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:27.324016 13015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:27.329587 13023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.329597 13021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.329691 13020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:27.329959 13015 server_base.cc:1034] running on GCE node
I20260925 12:14:27.330169 13015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:27.330417 13015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:27.331584 13015 hybrid_clock.cc:648] HybridClock initialized: now 1790338467331565 us; error 39 us; skew 500 ppm
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:27.334350 13015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:27.334795 13015 webserver.cc:460] Webserver started at http://127.0.98.3:46501/ using document root <none> and password file <none>
I20260925 12:14:27.335028 13015 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:27.335098 13015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:27.335253 13015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:27.350133 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "58f10213f7264e8f92fa9ef1b7e36347"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "170abd2d19a6ac85a05d038f797fd973"
server_key_iv: "d8921d7c90e90c8094aee43c250f480b"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.350625 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "58f10213f7264e8f92fa9ef1b7e36347"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "170abd2d19a6ac85a05d038f797fd973"
server_key_iv: "d8921d7c90e90c8094aee43c250f480b"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.354295 13015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 12:14:27.377825 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.378118 13015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 12:14:27.378214 13015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "58f10213f7264e8f92fa9ef1b7e36347"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "170abd2d19a6ac85a05d038f797fd973"
server_key_iv: "d8921d7c90e90c8094aee43c250f480b"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.378305 13015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:27.400625 13015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:27.403391 13015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:27.403630 13015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:27.403919 13015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:27.404328 13015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:27.404382 13015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.404413 13015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:27.404438 13015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.411767 13015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43945
I20260925 12:14:27.412245 13015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:14:27.413036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13015
I20260925 12:14:27.413136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:14:27.413326 392 external_mini_cluster.cc:1428] Setting key 3d209707338c86af8a7729a55355f359
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:27.424130 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.413357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35705 (local address 127.0.98.62:42643)
0925 12:14:27.413404 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:27.413406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:27.413613 (+ 207us) server_negotiation.cc:407] Connection header received
0925 12:14:27.414324 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:27.414325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:27.414338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:27.414396 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:27.415146 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.415966 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.417512 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.417842 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.417897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.420688 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:27.420690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:27.420690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:27.420699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:27.422433 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.422954 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.422955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.422956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.423014 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.423328 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.423329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.423329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.423476 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:27.423524 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:27.423943 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 12:14:27.424030 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:14:27.425227 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.413080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:35705)
0925 12:14:27.413472 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:27.413520 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:27.414189 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:27.414479 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:27.414484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:27.414745 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:27.415043 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.415049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.416079 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.416080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.417405 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.417408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.417990 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.417991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.418141 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.418822 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:27.418833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:27.420563 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:27.422532 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.422540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.422543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.422848 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.423086 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.423087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.423087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.423225 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.423574 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:27.423581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:27.423771 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 12:14:27.424289 (+ 518us) client_negotiation.cc:240] Negotiation successful
0925 12:14:27.424467 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":80,"threads_started":1}
I20260925 12:14:27.425688 13144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:27.425799 13144 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:27.426005 13144 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:27.426371 12660 ts_manager.cc:194] Registered new tserver with Master: 58f10213f7264e8f92fa9ef1b7e36347 (127.0.98.3:43945)
I20260925 12:14:27.426863 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35705
I20260925 12:14:27.428655 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43945 every 8 connection(s)
I20260925 12:14:27.436199 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:14:27.437067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 12741
I20260925 12:14:27.441901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 12880
I20260925 12:14:27.447291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13015
I20260925 12:14:27.453323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 12628
I20260925 12:14:27.458794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=35517
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:27.540220 13147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:27.540449 13147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:27.540472 13147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:27.541680 13147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:27.541729 13147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:27.541743 13147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:27.541755 13147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:27.541812 13147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:27.543447 13147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:27.543707 13147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:27.543951 13147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:27.549017 13155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.549023 13153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.549038 13152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:27.549090 13147 server_base.cc:1034] running on GCE node
I20260925 12:14:27.549499 13147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:27.549767 13147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:27.550915 13147 hybrid_clock.cc:648] HybridClock initialized: now 1790338467550894 us; error 36 us; skew 500 ppm
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:27.553448 13147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:27.553915 13147 webserver.cc:460] Webserver started at http://127.0.98.62:35517/ using document root <none> and password file <none>
I20260925 12:14:27.554143 13147 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:27.554191 13147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:27.617277 13147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:27.618075 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.618285 13147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:27.618399 13147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "eba9391c91e042018566bdc0a367d246"
format_stamp: "Formatted at 2026-09-25 12:14:25 on dist-test-slave-2hqq"
server_key: "30d777883472edf1f6ab58aaef8c05e6"
server_key_iv: "cb57184578ef534552b4b40312d3dcbe"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.618782 13147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:27.640998 13147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:27.642782 13147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:27.642946 13147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:27.647194 13147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260925 12:14:27.647230 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260925 12:14:27.647596 13147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:27.648785 13216 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:14:27.649448 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Bootstrap starting.
I20260925 12:14:27.654646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13147
I20260925 12:14:27.654944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35367
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=33037
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:27.657510 13216 log.cc:824] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:27.659253 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:14:27.659569 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Bootstrap complete.
I20260925 12:14:27.662042 13216 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:27.662299 13216 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba9391c91e042018566bdc0a367d246, State: Initialized, Role: FOLLOWER
I20260925 12:14:27.662510 13216 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:27.662612 13216 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:27.662664 13216 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:27.662714 13216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:14:27.663726 13216 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:27.663846 13216 leader_election.cc:302] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba9391c91e042018566bdc0a367d246; no voters:
I20260925 12:14:27.664049 13216 leader_election.cc:288] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:14:27.664085 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:14:27.664199 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [term 2 LEADER]: Becoming Leader. State: Replica: eba9391c91e042018566bdc0a367d246, State: Running, Role: LEADER
I20260925 12:14:27.664283 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260925 12:14:27.664316 13216 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:27.664579 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eba9391c91e042018566bdc0a367d246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260925 12:14:27.664662 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:27.664867 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba9391c91e042018566bdc0a367d246. Latest consensus state: current_term: 2 leader_uuid: "eba9391c91e042018566bdc0a367d246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba9391c91e042018566bdc0a367d246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260925 12:14:27.664983 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:27.665124 13224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:27.665659 13224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:27.665782 13224 catalog_manager.cc:1259] Loaded cluster ID: e89f02575e004b0e8ba75607071a49e3
I20260925 12:14:27.665817 13224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:27.665998 13216 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 12:14:27.669384 13224 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:27.669945 13224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Loaded TSK: 0
I20260925 12:14:27.670235 13224 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:14:27.767812 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:27.768110 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:27.768149 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:27.770068 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:27.770146 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:27.770191 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:27.772768 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33037
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:27.773052 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:27.773348 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:27.779908 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.780066 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:27.780282 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:27.780649 13218 server_base.cc:1034] running on GCE node
I20260925 12:14:27.780903 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:27.781217 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:27.782394 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790338467782380 us; error 31 us; skew 500 ppm
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:27.785454 13218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:27.786005 13218 webserver.cc:460] Webserver started at http://127.0.98.1:33037/ using document root <none> and password file <none>
I20260925 12:14:27.786264 13218 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:27.786360 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:27.810855 13218 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 12:14:27.812098 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.812309 13218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:27.812393 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "3618a79426e242d19681d3d70ad7a5c8"
format_stamp: "Formatted at 2026-09-25 12:14:26 on dist-test-slave-2hqq"
server_key: "d030c82fd489016b3043211d3a6f64b6"
server_key_iv: "2e29032f504a628753f4239285e91239"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:27.812898 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:27.853317 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:27.855584 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:27.855803 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:27.856115 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:27.856598 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:27.856689 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.856797 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:27.856866 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:27.864852 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35367
I20260925 12:14:27.865445 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:14:27.865978 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35367 every 8 connection(s)
I20260925 12:14:27.867856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13218
I20260925 12:14:27.868162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40557
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=33253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:14:27 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:27.879132 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.866260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:52499)
0925 12:14:27.866538 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:27.866582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:27.867146 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:27.867662 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:27.867666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:27.867890 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:27.868209 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.868216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.869893 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.869894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.871204 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:27.871207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.872182 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.872183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:27.872317 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.873003 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:27.873016 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:27.874783 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:27.877202 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.877209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.877212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.877512 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.877809 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:27.877809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:27.877810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:27.877951 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:27.878446 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:27.878453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:27.878617 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 12:14:27.878722 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 12:14:27.878879 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260925 12:14:27.880148 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:27.866317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52499 (local address 127.0.98.62:42643)
0925 12:14:27.866738 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:27.866746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:27.866782 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:14:27.867425 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:27.867430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:27.867476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:27.867618 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:27.868347 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.869763 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.871308 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:27.871947 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:27.872067 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:27.874935 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:27.874940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:27.874940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:27.874954 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:27.877106 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.877610 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.877614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.877614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.877731 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:27.878033 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:27.878034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:27.878035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:27.878336 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:27.878417 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:27.879808 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0925 12:14:27.879952 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":92,"threads_started":1}
I20260925 12:14:27.881178 13378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:27.881366 13378 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:27.881772 13378 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:27.882757 13378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:14:27.882822 13378 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:27.882880 13378 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:27.883302 13378 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:27.883386 13378 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:27.883692 13378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:14:27.988394 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:27.988804 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:27.988902 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:27.990824 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:27.990933 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:27.991021 13382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:27.993096 13382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33253
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:27.993378 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:27.993625 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:28.000077 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:28.000067 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:28.000468 13390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:28.000952 13382 server_base.cc:1034] running on GCE node
I20260925 12:14:28.001158 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:28.001412 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:28.002581 13382 hybrid_clock.cc:648] HybridClock initialized: now 1790338468002551 us; error 46 us; skew 500 ppm
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:28.005869 13382 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:28.006438 13382 webserver.cc:460] Webserver started at http://127.0.98.2:33253/ using document root <none> and password file <none>
I20260925 12:14:28.006713 13382 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:28.006810 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:28.024253 13382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 12:14:28.025110 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.025269 13382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:28.025354 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "0e8395a0e6da48ac4e618ec8e1d35e51"
server_key_iv: "deb4683551f953f4435ad69882fb5337"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:28.025806 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:28.044554 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:28.047029 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:28.047194 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:28.047534 13382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:28.047922 13382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:28.047978 13382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.048051 13382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:28.048081 13382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.054908 13382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40557
I20260925 12:14:28.055447 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:14:28.057994 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40557 every 8 connection(s)
I20260925 12:14:28.060735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13382
I20260925 12:14:28.061046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53223/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43945
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=46501
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:28.074091 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:28.058771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49439 (local address 127.0.98.62:42643)
0925 12:14:28.058820 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:28.058822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:28.059070 (+ 248us) server_negotiation.cc:407] Connection header received
0925 12:14:28.059904 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:28.059906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:28.059925 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:28.059987 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:28.060880 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.061538 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.063146 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.063496 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.063545 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:28.070883 (+ 7338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:28.070885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:28.070885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:28.070894 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:28.072574 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.073024 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.073026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.073026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.073086 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.073363 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.073364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.073364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.073485 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:28.073554 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:28.073901 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 12:14:28.073995 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:14:28.074275 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:28.058786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:49439)
0925 12:14:28.058946 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:28.058989 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:28.059737 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:28.060059 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:28.060063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:28.060364 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:28.060729 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:28.060735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.061639 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.061640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:28.063033 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:28.063037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.063678 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.063679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:28.063822 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:28.068963 (+ 5141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:28.068977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:28.070727 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:28.072663 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:28.072673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:28.072676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:28.072925 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:28.073154 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:28.073155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:28.073155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:28.073288 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:28.073556 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:28.073563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:28.073738 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 12:14:28.073847 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 12:14:28.074011 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 12:14:28.075183 13512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:28.075304 13512 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.075537 13512 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:28.076189 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:14:28.076285 13512 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.076349 13512 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:28.076686 13512 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.076766 13512 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:28.077062 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:14:28.173799 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:28.174080 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:28.174114 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:28.175957 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:28.176014 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:28.176054 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:28.178251 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:53223/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46501
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:28.178506 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:28.178754 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:28.185024 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:28.185286 13524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:28.185681 13515 server_base.cc:1034] running on GCE node
W20260925 12:14:28.185897 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:28.186136 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:28.186441 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:28.187633 13515 hybrid_clock.cc:648] HybridClock initialized: now 1790338468187610 us; error 37 us; skew 500 ppm
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:28.190814 13515 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:28.191351 13515 webserver.cc:460] Webserver started at http://127.0.98.3:46501/ using document root <none> and password file <none>
I20260925 12:14:28.191635 13515 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:28.191728 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:28.208292 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 12:14:28.209230 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.209435 13515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:28.209532 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "58f10213f7264e8f92fa9ef1b7e36347"
format_stamp: "Formatted at 2026-09-25 12:14:27 on dist-test-slave-2hqq"
server_key: "170abd2d19a6ac85a05d038f797fd973"
server_key_iv: "d8921d7c90e90c8094aee43c250f480b"
server_key_version: "kuduclusterkey@0"
I20260925 12:14:28.210028 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:28.237530 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:28.240018 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:28.240190 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:28.240482 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:28.240878 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:28.240932 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.240964 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:28.241041 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:28.248368 13515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43945
I20260925 12:14:28.248440 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43945 every 8 connection(s)
I20260925 12:14:28.248785 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:28.257753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13515
I20260925 12:14:28.258718 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:28.249763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56157 (local address 127.0.98.62:42643)
0925 12:14:28.249812 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:28.249815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:28.250080 (+ 265us) server_negotiation.cc:407] Connection header received
0925 12:14:28.250680 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:28.250681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:28.250692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:28.250751 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:28.251330 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.251827 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.252964 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.253177 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.253267 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:28.255582 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:28.255584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:28.255584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:28.255592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:28.257096 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.257570 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.257571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.257571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.257631 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.257913 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.257914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.257914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.258042 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:28.258085 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:28.258499 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 12:14:28.258588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:14:28.259418 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:28.249785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:56157)
0925 12:14:28.249927 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:28.249968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:28.250562 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:28.250838 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:28.250842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:28.251020 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:28.251219 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:28.251223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.251918 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.251920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:28.252873 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:28.252875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.253287 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.253288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:28.253452 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:28.253992 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:28.254000 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:28.255482 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:28.257186 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:28.257192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:28.257194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:28.257475 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:28.257700 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:28.257701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:28.257701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:28.257835 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:28.258157 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:28.258163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:28.258340 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 12:14:28.259036 (+ 696us) client_negotiation.cc:240] Negotiation successful
0925 12:14:28.259192 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 12:14:28.260053 13646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260925 12:14:28.260142 13646 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.260320 13646 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:28.261010 13646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 12:14:28.261091 13646 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.261163 13646 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:28.261545 13646 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.261621 13646 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260925 12:14:28.261937 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:14:28.268038 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:28.261264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.62:42643)
0925 12:14:28.261302 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:28.261303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:28.261754 (+ 451us) server_negotiation.cc:407] Connection header received
0925 12:14:28.261777 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:28.261778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:28.261788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:28.261835 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:28.262549 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.263011 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.263782 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:28.263984 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:28.264068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:28.265458 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:28.265459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:28.265459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:28.265465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:28.266910 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.267276 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.267277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.267278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.267337 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:28.267533 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:28.267534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:28.267534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:28.267645 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:28.267816 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:28.267856 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 12:14:28.267931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 12:14:28 dist-test-slave-2hqq krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:28.883862 13378 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:28.884038 13378 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:28.884440 13180 ts_manager.cc:194] Registered new tserver with Master: 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367)
I20260925 12:14:28.885479 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52499
I20260925 12:14:29.077226 13512 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:29.077387 13512 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:29.077797 13180 ts_manager.cc:194] Registered new tserver with Master: bf1f8ac5e1f341f7b1f81bddd98784c4 (127.0.98.2:40557)
I20260925 12:14:29.078339 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49439
Sep 25 12:14:29 dist-test-slave-2hqq krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 12:14:29.262120 13646 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:29.262323 13646 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260925 12:14:29.262650 13180 ts_manager.cc:194] Registered new tserver with Master: 58f10213f7264e8f92fa9ef1b7e36347 (127.0.98.3:43945)
I20260925 12:14:29.263199 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56157
I20260925 12:14:29.263501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:14:29.272480 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.264460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.0.98.62:42643)
0925 12:14:29.264617 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.264619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.264697 (+ 78us) server_negotiation.cc:407] Connection header received
0925 12:14:29.264807 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.264808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.264821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.264911 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:29.265548 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.266363 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.267411 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.267780 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.267849 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.269350 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:29.269351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:29.269351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:29.269358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:29.271197 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:29.271642 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:29.271643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:29.271644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:29.271717 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:29.271937 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:29.271938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:29.271938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:29.272069 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:29.272219 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.272300 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.272380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260925 12:14:29.274940 13179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:14:29.277137 13179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:14:29.285524 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40557 (local address 127.0.0.1:54092)
0925 12:14:29.281065 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.281074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.281123 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.281402 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:29.281405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:29.281407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:29.281475 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.281477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.282757 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.282758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.284165 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.284168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.285222 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.285222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.285301 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.285316 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 12:14:29.285387 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:14:29.285435 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":45,"thread_start_us":84,"threads_started":1}
I20260925 12:14:29.285739 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43945 (local address 127.0.0.1:42086)
0925 12:14:29.280799 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.280838 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.280902 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.281420 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:29.281421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:29.281423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:29.281478 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.281480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.282760 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.282761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.284166 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.284169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.285518 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.285519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.285581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.285585 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:29.285624 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 12:14:29.285662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":77,"thread_start_us":94,"threads_started":1}
I20260925 12:14:29.286187 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.0.98.2:40557)
0925 12:14:29.281024 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.281029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.281151 (+ 122us) server_negotiation.cc:407] Connection header received
0925 12:14:29.281203 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.281206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.281222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.281310 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:29.281564 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.282658 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.284274 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.285027 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.285098 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.285172 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.286055 (+ 883us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.286098 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":53,"threads_started":1}
I20260925 12:14:29.286948 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.98.3:43945)
0925 12:14:29.281220 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.281224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.281233 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:14:29.281267 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.281269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.281283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.281348 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:29.281549 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.282662 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.284260 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.285419 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.285806 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.286180 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.286207 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.286872 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":45,"threads_started":1}
I20260925 12:14:29.286978 13446 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 (DEFAULT_TABLE table=test-table [id=5717f4e91eef408588da2bfc146271f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:29.287341 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0e5f5914448b599327401ec1ddaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:29.287654 13580 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 (DEFAULT_TABLE table=test-table [id=5717f4e91eef408588da2bfc146271f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:29.287952 13580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0e5f5914448b599327401ec1ddaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:29.291288 13678 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Bootstrap starting.
I20260925 12:14:29.291985 13678 tablet_bootstrap.cc:654] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:29.292348 13678 log.cc:824] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:29.292665 13679 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Bootstrap starting.
I20260925 12:14:29.293164 13678 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: No bootstrap required, opened a new log
I20260925 12:14:29.293241 13678 ts_tablet_manager.cc:1397] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:29.293460 13679 tablet_bootstrap.cc:654] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:29.294090 13679 log.cc:824] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:29.294997 13679 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: No bootstrap required, opened a new log
I20260925 12:14:29.295058 13679 ts_tablet_manager.cc:1397] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:29.295837 13678 raft_consensus.cc:348] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.295997 13678 raft_consensus.cc:374] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:29.296033 13678 raft_consensus.cc:729] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1f8ac5e1f341f7b1f81bddd98784c4, State: Initialized, Role: FOLLOWER
I20260925 12:14:29.296355 13678 consensus_queue.cc:260] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.296749 13678 ts_tablet_manager.cc:1428] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:14:29.296970 13512 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260925 12:14:29.297560 13679 raft_consensus.cc:348] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.297724 13679 raft_consensus.cc:374] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:29.297755 13679 raft_consensus.cc:729] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f10213f7264e8f92fa9ef1b7e36347, State: Initialized, Role: FOLLOWER
I20260925 12:14:29.297861 13679 consensus_queue.cc:260] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.298095 13679 ts_tablet_manager.cc:1428] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:14:29.298153 13646 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260925 12:14:29.304867 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35367 (local address 127.0.0.1:57528)
0925 12:14:29.285938 (+ 5475us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.285947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.285982 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.300944 (+ 14962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:29.300946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:29.300948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:29.301001 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.301003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.302458 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.302459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.303577 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.303580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.304605 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.304606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.304668 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.304673 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:14:29.304732 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:14:29.304778 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5464,"thread_start_us":80,"threads_started":1}
I20260925 12:14:29.304872 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.280579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.98.1:35367)
0925 12:14:29.300730 (+ 20151us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.300735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.300749 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:14:29.300789 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.300791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.300808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.301073 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:29.301235 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.302369 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.303682 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.304510 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.304595 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.304677 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.304748 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.304791 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20105,"thread_start_us":51,"threads_started":1}
I20260925 12:14:29.305630 13312 tablet_service.cc:1467] Processing CreateTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 (DEFAULT_TABLE table=test-table [id=5717f4e91eef408588da2bfc146271f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:29.305966 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d0e5f5914448b599327401ec1ddaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:29.307430 13682 raft_consensus.cc:482] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:29.307569 13682 raft_consensus.cc:504] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.308346 13682 leader_election.cc:288] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f10213f7264e8f92fa9ef1b7e36347 (127.0.98.3:43945), 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367)
W20260925 12:14:29.309329 13513 tablet.cc:2369] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:14:29.309927 13687 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: Bootstrap starting.
I20260925 12:14:29.310600 13687 tablet_bootstrap.cc:654] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:29.310952 13687 log.cc:824] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:29.312438 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.308278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50941 (local address 127.0.98.3:43945)
0925 12:14:29.308332 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.308334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.308718 (+ 384us) server_negotiation.cc:407] Connection header received
0925 12:14:29.308822 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.308823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.308835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.308866 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:29.309098 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.309775 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.311168 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.311751 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.311800 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.311810 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.312322 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.312372 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 12:14:29.312829 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.308092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43945 (local address 127.0.98.2:50941)
0925 12:14:29.308649 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.308662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.308746 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.308930 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:29.308931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:29.308933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:29.309018 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.309019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.309870 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.309871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.311064 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.311067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.312145 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.312146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.312233 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.312237 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:29.312579 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 12:14:29.312738 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":103,"thread_start_us":92,"threads_started":1}
I20260925 12:14:29.313158 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0e5f5914448b599327401ec1ddaa1" candidate_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f10213f7264e8f92fa9ef1b7e36347" is_pre_election: true
I20260925 12:14:29.313323 13584 raft_consensus.cc:2454] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1f8ac5e1f341f7b1f81bddd98784c4 in term 0.
I20260925 12:14:29.313553 13402 leader_election.cc:302] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f10213f7264e8f92fa9ef1b7e36347, bf1f8ac5e1f341f7b1f81bddd98784c4; no voters:
I20260925 12:14:29.313728 13682 raft_consensus.cc:2781] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:14:29.313823 13682 raft_consensus.cc:482] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:14:29.313897 13682 raft_consensus.cc:3037] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:29.314810 13682 raft_consensus.cc:504] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.315060 13682 leader_election.cc:288] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 election: Requested vote from peers 58f10213f7264e8f92fa9ef1b7e36347 (127.0.98.3:43945), 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367)
I20260925 12:14:29.315184 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0e5f5914448b599327401ec1ddaa1" candidate_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f10213f7264e8f92fa9ef1b7e36347"
I20260925 12:14:29.315264 13584 raft_consensus.cc:3037] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:29.316102 13584 raft_consensus.cc:2454] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1f8ac5e1f341f7b1f81bddd98784c4 in term 1.
I20260925 12:14:29.316285 13402 leader_election.cc:302] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f10213f7264e8f92fa9ef1b7e36347, bf1f8ac5e1f341f7b1f81bddd98784c4; no voters:
I20260925 12:14:29.316421 13682 raft_consensus.cc:2781] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:29.316587 13682 raft_consensus.cc:686] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 1 LEADER]: Becoming Leader. State: Replica: bf1f8ac5e1f341f7b1f81bddd98784c4, State: Running, Role: LEADER
I20260925 12:14:29.316679 13682 consensus_queue.cc:237] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.317521 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.308656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39871 (local address 127.0.98.1:35367)
0925 12:14:29.309805 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.309806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.311152 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 12:14:29.311236 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.311236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.311245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.311270 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:29.311618 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.312261 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.314422 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.314993 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.316388 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.316399 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.317424 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.317462 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":38,"threads_started":1}
I20260925 12:14:29.317579 13180 catalog_manager.cc:5495] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1f8ac5e1f341f7b1f81bddd98784c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } health_report { overall_health: UNKNOWN } } }
I20260925 12:14:29.317790 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.308469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35367 (local address 127.0.98.2:39871)
0925 12:14:29.311076 (+ 2607us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.311101 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.311175 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.311320 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:29.311321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:29.311322 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:29.311381 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.311382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.312352 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.312353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.314333 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:29.314336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.317275 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.317276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:29.317343 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.317348 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:14:29.317656 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 12:14:29.317700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"thread_start_us":110,"threads_started":1}
I20260925 12:14:29.318714 13687 tablet_bootstrap.cc:492] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: No bootstrap required, opened a new log
I20260925 12:14:29.318795 13687 ts_tablet_manager.cc:1397] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260925 12:14:29.318121 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0e5f5914448b599327401ec1ddaa1" candidate_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3618a79426e242d19681d3d70ad7a5c8" is_pre_election: true
I20260925 12:14:29.319898 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d0e5f5914448b599327401ec1ddaa1" candidate_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3618a79426e242d19681d3d70ad7a5c8"
W20260925 12:14:29.319977 13399 leader_election.cc:341] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367): Illegal state: must be running to vote when last-logged opid is not known
W20260925 12:14:29.320088 13399 leader_election.cc:341] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367): Illegal state: must be running to vote when last-logged opid is not known
I20260925 12:14:29.320696 13687 raft_consensus.cc:348] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.320874 13687 raft_consensus.cc:374] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:29.320922 13687 raft_consensus.cc:729] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3618a79426e242d19681d3d70ad7a5c8, State: Initialized, Role: FOLLOWER
I20260925 12:14:29.321076 13687 consensus_queue.cc:260] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f8ac5e1f341f7b1f81bddd98784c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40557 } } peers { permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 } } peers { permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 } }
I20260925 12:14:29.321323 13687 ts_tablet_manager.cc:1428] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:29.321468 13378 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260925 12:14:29.332350 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.329121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.0.98.2:40557)
0925 12:14:29.329166 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:29.329168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:29.329176 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:14:29.329211 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:29.329212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:29.329231 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:29.329311 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:14:29.329515 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.330350 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.331442 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:29.331717 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:29.331784 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:29.331968 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:14:29.332100 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:14:29.332178 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:29.332257 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:14:29.332278 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:14:29.333985 13332 raft_consensus.cc:3037] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:29.334152 13584 raft_consensus.cc:1261] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 1 FOLLOWER]: Refusing update from remote peer bf1f8ac5e1f341f7b1f81bddd98784c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:29.334474 13682 consensus_queue.cc:1035] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f10213f7264e8f92fa9ef1b7e36347" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:29.335002 13332 raft_consensus.cc:1261] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 1 FOLLOWER]: Refusing update from remote peer bf1f8ac5e1f341f7b1f81bddd98784c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:29.335248 13682 consensus_queue.cc:1035] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3618a79426e242d19681d3d70ad7a5c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:29.337623 13693 mvcc.cc:205] Tried to move back new op lower bound from 7333226370389921792 to 7333226370322108416. Current Snapshot: MvccSnapshot[applied={T|T < 7333226370389921792}]
I20260925 12:14:29.337963 13696 mvcc.cc:205] Tried to move back new op lower bound from 7333226370389921792 to 7333226370322108416. Current Snapshot: MvccSnapshot[applied={T|T < 7333226370389921792}]
I20260925 12:14:29.342290 13694 mvcc.cc:205] Tried to move back new op lower bound from 7333226370389921792 to 7333226370322108416. Current Snapshot: MvccSnapshot[applied={T|T < 7333226370389921792}]
I20260925 12:14:29.343989 13180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:29.344087 13180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:29.344852 13180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eba9391c91e042018566bdc0a367d246: Sending DeleteTablet for 3 replicas of tablet c6d0e5f5914448b599327401ec1ddaa1
I20260925 12:14:29.345290 13580 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42086
I20260925 12:14:29.345449 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260925 12:14:29.345686 13703 tablet_replica.cc:330] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8: stopping tablet replica
I20260925 12:14:29.345826 13703 raft_consensus.cc:2229] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:29.345852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13218
I20260925 12:14:29.346011 13703 raft_consensus.cc:2258] T c6d0e5f5914448b599327401ec1ddaa1 P 3618a79426e242d19681d3d70ad7a5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:29.347456 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d0e5f5914448b599327401ec1ddaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54092
W20260925 12:14:29.351215 13164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35367 (error 108)
W20260925 12:14:29.356557 13673 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:14:29.351608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35367 (local address 127.0.0.1:57536)
0925 12:14:29.351650 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.351661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:29.351697 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:29.356509 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 12:14:29.356608 13704 tablet_replica.cc:330] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: stopping tablet replica
I20260925 12:14:29.356724 13704 raft_consensus.cc:2229] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 1 LEADER]: Raft consensus shutting down.
W20260925 12:14:29.356797 13164 catalog_manager.cc:4584] TS 3618a79426e242d19681d3d70ad7a5c8 (127.0.98.1:35367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d0e5f5914448b599327401ec1ddaa1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:14:29.356854 13704 raft_consensus.cc:2258] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:29.357265 13702 tablet_replica.cc:330] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: stopping tablet replica
I20260925 12:14:29.357350 13704 ts_tablet_manager.cc:1905] T c6d0e5f5914448b599327401ec1ddaa1 P bf1f8ac5e1f341f7b1f81bddd98784c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:29.357430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13382
I20260925 12:14:29.358584 13702 raft_consensus.cc:2229] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:29.358711 13702 raft_consensus.cc:2258] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:29.359318 13702 ts_tablet_manager.cc:1905] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:29.361253 13702 ts_tablet_manager.cc:1918] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:29.361361 13702 log.cc:1196] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/c6d0e5f5914448b599327401ec1ddaa1
I20260925 12:14:29.361611 13702 ts_tablet_manager.cc:1939] T c6d0e5f5914448b599327401ec1ddaa1 P 58f10213f7264e8f92fa9ef1b7e36347: Deleting consensus metadata
I20260925 12:14:29.362056 13167 catalog_manager.cc:4841] TS 58f10213f7264e8f92fa9ef1b7e36347 (127.0.98.3:43945): tablet c6d0e5f5914448b599327401ec1ddaa1 (table test-table [id=5717f4e91eef408588da2bfc146271f1]) successfully deleted
I20260925 12:14:29.363765 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:29.363621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40557 (local address 127.0.0.1:54112)
0925 12:14:29.363657 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:29.363727 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 12:14:29.364641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13515
I20260925 12:14:29.369865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13147
I20260925 12:14:29.374944 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 12:14:29.708969 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 12:14:29.709059 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 12:14:30.053524 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 12:14:30.053 UTC [12214] LOG: received smart shutdown request
2026-09-25 12:14:30.057 UTC [12214] LOG: background worker "logical replication launcher" (PID 12223) exited with exit code 1
2026-09-25 12:14:30.057 UTC [12218] LOG: shutting down
2026-09-25 12:14:30.233 UTC [12214] LOG: database system is shut down
2026-09-25T12:14:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48180 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:14:30 dist-test-slave-2hqq krb5kdc[13714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:14:30 dist-test-slave-2hqq krb5kdc[13714](info): set up 2 sockets
Sep 25 12:14:30 dist-test-slave-2hqq krb5kdc[13714](info): commencing operation
krb5kdc: starting...
W20260925 12:14:31.372416 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:14:31.491550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:31.573637 13727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:31.573853 13727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:31.573875 13727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:31.575078 13727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:31.575124 13727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:31.575138 13727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:31.575150 13727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:31.575162 13727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:31.576740 13727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:31.577008 13727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:31.577253 13727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:31.582554 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.582576 13733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.582578 13735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:31.582787 13727 server_base.cc:1034] running on GCE node
I20260925 12:14:31.583132 13727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:31.583448 13727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:31.584576 13727 hybrid_clock.cc:648] HybridClock initialized: now 1790338471584555 us; error 36 us; skew 500 ppm
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:31.591696 13727 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:31.592173 13727 webserver.cc:460] Webserver started at http://127.0.98.62:41779/ using document root <none> and password file <none>
I20260925 12:14:31.592425 13727 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:31.592492 13727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:31.592648 13727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:31.593720 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "365fbfd53fea408fb21ac0f306fd8000"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "414cc7cec49e39efc53a4d2d2d6c8286"
server_key_iv: "5312bac4875aaffbc737ab4452e13da7"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.594074 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "365fbfd53fea408fb21ac0f306fd8000"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "414cc7cec49e39efc53a4d2d2d6c8286"
server_key_iv: "5312bac4875aaffbc737ab4452e13da7"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.595417 13727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:14:31.596120 13742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:31.596299 13727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:14:31.596416 13727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "365fbfd53fea408fb21ac0f306fd8000"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "414cc7cec49e39efc53a4d2d2d6c8286"
server_key_iv: "5312bac4875aaffbc737ab4452e13da7"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.596491 13727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:31.617801 13727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:31.619871 13727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:31.620059 13727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:31.624181 13727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39481
I20260925 12:14:31.624229 13794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39481 every 8 connection(s)
I20260925 12:14:31.624630 13727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:31.625538 13795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:31.626585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13727
I20260925 12:14:31.626660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:14:31.626850 392 external_mini_cluster.cc:1428] Setting key 6b66ede4eeb413c5ef1067070746a8ac
I20260925 12:14:31.627218 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 12:14:31.638449 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.628143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.98.62:39481)
0925 12:14:31.628273 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.628278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.628316 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:14:31.629039 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.629041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.629057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.629137 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.629888 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.630828 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.632252 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.632623 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.632711 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.634991 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.635006 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.635006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.635013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.636703 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.637162 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.637165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.637166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.637282 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.637520 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.637522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.637522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.637756 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.637855 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.638039 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.638184 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 12:14:31.638458 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.627925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.0.1:39824)
0925 12:14:31.628136 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.628188 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.628883 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.629183 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.629186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.629567 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.629792 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.629798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.630920 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.630921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.632169 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.632173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.632703 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.632704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.632785 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.633276 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.633286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.634910 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.636821 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.636827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.636829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.637059 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.637356 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.637357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.637357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.637452 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.637831 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.637836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.637901 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.638010 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.638256 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":96,"thread_start_us":92,"threads_started":1}
I20260925 12:14:31.639542 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:14:31.639968 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.639791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35308)
0925 12:14:31.639837 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.639881 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:14:31.640120 13744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
I20260925 12:14:31.640620 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.640533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35314)
0925 12:14:31.640567 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.640599 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 12:14:31.640726 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:14:31.651778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44341
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:31.689960 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } attempt: 1
I20260925 12:14:31.690523 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.690375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35320)
0925 12:14:31.690435 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.690486 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:14:31.690933 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.690845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35336)
0925 12:14:31.690885 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.690912 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 12:14:31.691064 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
W20260925 12:14:31.733067 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:31.733273 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:31.733294 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:31.734436 13802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:31.734479 13802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:31.734500 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:31.734514 13802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:31.734526 13802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:31.736027 13802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:31.736276 13802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:31.736521 13802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:31.741317 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.741339 13807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.741312 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:31.741621 13802 server_base.cc:1034] running on GCE node
I20260925 12:14:31.741855 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:31.742229 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:31.743397 13802 hybrid_clock.cc:648] HybridClock initialized: now 1790338471743382 us; error 42 us; skew 500 ppm
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:31.745980 13802 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:14:31.746428 13802 webserver.cc:460] Webserver started at http://127.0.98.61:40217/ using document root <none> and password file <none>
I20260925 12:14:31.746649 13802 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:31.746712 13802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:31.746867 13802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:31.747934 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data/instance:
uuid: "0672b7d4205a4fc58b0f42bb13edf087"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "3f702df57764e8b99e5a5e7fa17c4b15"
server_key_iv: "ca9b8c3fceec38e33220f69bea0defa3"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.748306 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal/instance:
uuid: "0672b7d4205a4fc58b0f42bb13edf087"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "3f702df57764e8b99e5a5e7fa17c4b15"
server_key_iv: "ca9b8c3fceec38e33220f69bea0defa3"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.749650 13802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:31.750378 13817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:31.750550 13802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:14:31.750644 13802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "0672b7d4205a4fc58b0f42bb13edf087"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "3f702df57764e8b99e5a5e7fa17c4b15"
server_key_iv: "ca9b8c3fceec38e33220f69bea0defa3"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.750725 13802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:31.759235 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } attempt: 2
I20260925 12:14:31.759657 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.759533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35342)
0925 12:14:31.759590 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.759626 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:14:31.760103 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.759952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35358)
0925 12:14:31.760032 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.760072 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 12:14:31.760247 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44341: connect: Connection refused (error 111)
I20260925 12:14:31.761477 13802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:31.763773 13802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:31.763890 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:31.767577 13802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44341
I20260925 12:14:31.767609 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44341 every 8 connection(s)
I20260925 12:14:31.767964 13802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:14:31.768893 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:31.770440 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:31.776373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13802
I20260925 12:14:31.776468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-1/wal/instance
I20260925 12:14:31.776732 392 external_mini_cluster.cc:1428] Setting key 155a07df5d4ec293b47074558b56613f
I20260925 12:14:31.779282 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.771131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.98.62:39481)
0925 12:14:31.771190 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.771192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.771383 (+ 191us) server_negotiation.cc:407] Connection header received
0925 12:14:31.771843 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.771844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.771855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.771899 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.772474 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.773019 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.773953 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.774156 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.774205 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.776223 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.776225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.776225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.776233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.777920 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.778303 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.778305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.778305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.778369 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.778605 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.778606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.778607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.778731 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.778831 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.779110 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.779196 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:14:31.779474 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.771104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.0.1:39828)
0925 12:14:31.771254 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.771296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.771750 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.771965 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.771968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.772207 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.772403 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.772408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.773097 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.773098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.773887 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.773888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.774221 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.774222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.774318 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.774818 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.774827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.776112 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.778001 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.778007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.778009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.778218 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.778435 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.778436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.778437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.778539 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.778825 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.778832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.778976 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.779077 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.779229 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260925 12:14:31.780129 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 12:14:31.787768 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.780388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35364)
0925 12:14:31.780441 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.780455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.780533 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.780750 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.780751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.780907 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.780974 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.780975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.781804 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.781805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.782666 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.782669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.783113 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.783114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.783193 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.783567 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.783572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.784820 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.786597 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.786598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.786598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.786756 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.787061 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.787062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.787062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.787117 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.787423 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.787424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.787460 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.787541 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.787638 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:14:31.787801 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.780439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.0.98.61:44341)
0925 12:14:31.780608 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.780613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.780631 (+ 18us) server_negotiation.cc:407] Connection header received
0925 12:14:31.780657 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.780659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.780669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.780730 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.781062 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.781725 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.782748 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.783039 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.783095 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.784916 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.784923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.784923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.784931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.786525 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.786916 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.786919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.786919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.786990 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.787178 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.787179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.787180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.787328 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.787438 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.787576 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.787688 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":142,"thread_start_us":117,"threads_started":1}
I20260925 12:14:31.788420 13870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:14:31.788790 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.788662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47734)
0925 12:14:31.788708 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.788759 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:14:31.788975 13821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
I20260925 12:14:31.789412 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.789316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47742)
0925 12:14:31.789360 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.789386 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:14:31.789505 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38853: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:14:31.802946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38853
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:31.838738 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } attempt: 1
I20260925 12:14:31.839346 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.839175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47744)
0925 12:14:31.839241 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.839295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:14:31.839728 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.839634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47756)
0925 12:14:31.839680 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.839708 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:14:31.839840 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38853: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
I20260925 12:14:31.851424 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } attempt: 3
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:14:31.858497 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.851841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.98.61:44341)
0925 12:14:31.851906 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.851909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.851919 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:14:31.852025 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.852025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.852037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.852095 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.852407 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.853059 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.853776 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.854032 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.854123 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.855919 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.855921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.855921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.855928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.857395 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.857732 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.857733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.857734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.857789 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.857989 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.857990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.857990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.858101 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.858173 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.858320 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.858400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:14:31.858505 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.851785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35368)
0925 12:14:31.851835 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.851851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.851918 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.852114 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.852114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.852272 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.852333 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.852335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.853136 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.853138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.853692 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.853694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.854123 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.854124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.854205 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.854670 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.854675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.855834 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.857495 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.857496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.857497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.857663 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.857851 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.857852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.857852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.857908 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.858165 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.858166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.858215 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.858297 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.858400 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 12:14:31.858902 13795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:14:31.859266 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.859172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47760)
0925 12:14:31.859218 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.859247 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 12:14:31.859565 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.859478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47768)
0925 12:14:31.859523 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.859546 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 12:14:31.859652 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38853: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
W20260925 12:14:31.886229 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:31.886432 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:31.886454 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:31.887568 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:31.887612 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:31.887626 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:31.887638 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:31.887650 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:31.889218 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39481,127.0.98.61:44341,127.0.98.60:38853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:31.889469 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:31.889695 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:14:31.890818 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } attempt: 1
I20260925 12:14:31.891301 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.891181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47772)
0925 12:14:31.891236 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.891276 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:14:31.891630 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.891542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47788)
0925 12:14:31.891582 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.891608 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 12:14:31.891712 13795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38853: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
W20260925 12:14:31.894654 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.894657 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:31.894658 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:31.894843 13877 server_base.cc:1034] running on GCE node
I20260925 12:14:31.895143 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:31.895421 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:31.896538 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790338471896533 us; error 34 us; skew 500 ppm
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:31.898980 13877 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:14:31.899366 13877 webserver.cc:460] Webserver started at http://127.0.98.60:33933/ using document root <none> and password file <none>
I20260925 12:14:31.899587 13877 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:31.899633 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:31.899724 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:31.900882 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data/instance:
uuid: "a6596e6e1c604ace9fca1be7291747a8"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "f38bdddf92dabf8195513243e97c8606"
server_key_iv: "0a00215be92b42a8197a618eda5cb9c0"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.901180 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal/instance:
uuid: "a6596e6e1c604ace9fca1be7291747a8"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "f38bdddf92dabf8195513243e97c8606"
server_key_iv: "0a00215be92b42a8197a618eda5cb9c0"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.902450 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:14:31.903214 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:31.903385 13877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:31.903494 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "a6596e6e1c604ace9fca1be7291747a8"
format_stamp: "Formatted at 2026-09-25 12:14:31 on dist-test-slave-2hqq"
server_key: "f38bdddf92dabf8195513243e97c8606"
server_key_iv: "0a00215be92b42a8197a618eda5cb9c0"
server_key_version: "encryptionkey@0"
I20260925 12:14:31.903566 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:31.908011 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } attempt: 2
I20260925 12:14:31.908449 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.908323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47796)
0925 12:14:31.908382 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.908419 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:14:31.908810 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.908718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47802)
0925 12:14:31.908760 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.908787 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 12:14:31.908919 13870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38853: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38853: connect: Connection refused (error 111)
I20260925 12:14:31.923802 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:31.925738 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:31.925855 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:31.929965 13877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38853
I20260925 12:14:31.930001 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38853 every 8 connection(s)
I20260925 12:14:31.930356 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:14:31.931358 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:31.933157 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:14:31.937721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 13877
I20260925 12:14:31.937834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790338290997044-392-0/minicluster-data/master-2/wal/instance
I20260925 12:14:31.938038 392 external_mini_cluster.cc:1428] Setting key d9a1f7f5b8f095abbf7b1869c356ac2c
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:31.943106 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.934042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.98.62:39481)
0925 12:14:31.934109 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.934111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.934316 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:14:31.934796 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.934797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.934807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.934858 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.935497 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.935959 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.937139 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.937322 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.937378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.940166 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.940168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.940168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.940177 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.941741 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.942147 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.942149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.942150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.942210 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.942451 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.942452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.942452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.942573 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.942639 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.942919 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.943013 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:14:31.943344 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.934034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39481 (local address 127.0.0.1:39834)
0925 12:14:31.934189 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.934228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.934714 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.934925 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.934929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.935200 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.935416 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.935420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.936059 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.936060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.937069 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.937071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.937413 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.937414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.937531 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.938271 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.938283 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.940079 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.941817 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.941825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.941827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.942057 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.942274 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.942275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.942275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.942379 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.942641 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.942648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.942784 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.942898 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.943051 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:14:31.944016 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:14:31.947395 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.938467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.98.60:38853)
0925 12:14:31.938662 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.938667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.938723 (+ 56us) server_negotiation.cc:407] Connection header received
0925 12:14:31.938754 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.938756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.938778 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.938879 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.939589 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.940476 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.941577 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.941907 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.942014 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.944603 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.944611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.944611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.944618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.946228 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.946573 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.946576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.946576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.946634 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.946825 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.946826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.946826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.946971 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.947109 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.947204 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.947277 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:14:31.951439 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.944284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44341 (local address 127.0.0.1:35380)
0925 12:14:31.944338 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.944350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.944397 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.944566 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.944567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.944744 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.944790 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.944791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.945859 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.945860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.946383 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.946384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.946860 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.946861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.946942 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.947360 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.947365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.948725 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.950503 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.950504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.950504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.950658 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.950864 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.950864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.950865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.950924 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.951168 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.951168 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.951202 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.951253 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.951335 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:14:31.951458 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.944375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.0.98.61:44341)
0925 12:14:31.944423 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.944425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.944435 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:14:31.944462 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.944463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.944473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.944561 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.944971 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.945733 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.946507 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.946783 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.946866 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.948806 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.948808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.948808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.948817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.950411 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.950731 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.950732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.950733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.950802 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.950982 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.950983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.950983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.951105 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.951172 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.951285 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.951386 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:14:31.951985 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:14:31.957816 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.948344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.0.98.62:39481)
0925 12:14:31.948384 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.948385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.948417 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:14:31.948438 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.948439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.948449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.948522 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.949116 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.949734 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.950774 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.951048 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.951100 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.954192 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.954193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.954194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.954201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.955976 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.956405 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.956406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.956407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.956503 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.956730 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.956731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.956731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.956879 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.957560 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.957615 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.957700 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:14:31.959920 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.952290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47822)
0925 12:14:31.952339 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.952349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.952387 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.952547 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.952548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.952661 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.952688 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.952689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.953478 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.953479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.954182 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.954185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.954683 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.954684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.954732 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.955097 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.955102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.956474 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.958893 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.958893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.958894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.959100 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.959289 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.959290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.959290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.959345 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.959615 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.959616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.959656 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.959716 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.959803 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:14:31.960211 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.952338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.0.98.60:38853)
0925 12:14:31.952380 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.952383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.952393 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:14:31.952437 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.952438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.952448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.952543 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.952756 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.953361 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.954267 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.954625 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.954686 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.956600 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.956601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.956602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.956609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.958808 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.959177 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.959178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.959178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.959230 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.959406 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.959406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.959407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.959532 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.959593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.960067 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.960138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:14:31.962251 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: Bootstrap starting.
I20260925 12:14:31.963230 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:31.963531 13945 log.cc:824] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: Log is configured to *not* fsync() on all Append() calls
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:14:31.964340 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: No bootstrap required, opened a new log
I20260925 12:14:31.966566 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:31.966753 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:31.966790 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6596e6e1c604ace9fca1be7291747a8, State: Initialized, Role: FOLLOWER
I20260925 12:14:31.966907 13795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } attempt: 2
I20260925 12:14:31.966939 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:31.967506 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.959062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.0.98.61:44341)
0925 12:14:31.959101 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.959103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.959110 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:14:31.959156 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.959157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.959167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.959240 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.959906 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.960907 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.961830 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.962123 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.962201 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.964723 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.964725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.964725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.964732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.966306 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.966629 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.966630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.966630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.966687 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.966928 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.966929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.966930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.967073 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.967270 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.967307 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.967413 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:14:31.967602 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:31.967671 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:31.968042 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:31.969197 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 12:14:31.969897 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:14:31.970010 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 12:14:31 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:14:31.971272 13877 master_runner.cc:186] Error getting master registration for 127.0.98.62:39481: OK, instance_id { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" instance_seqno: 1790338471622971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:14:31.974254 13877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:14:31.974650 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.967140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47834)
0925 12:14:31.967187 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:31.967196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:31.967235 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:31.967499 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:31.967500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:31.967665 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:31.967712 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.967714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.968577 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.968578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.969429 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:31.969432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.969881 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.969882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:31.969941 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.970415 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:31.970420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:31.971779 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:31.973580 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.973581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.973581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.973786 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.973998 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:31.973999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:31.973999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:31.974056 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:31.974351 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:31.974351 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:31.974397 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 12:14:31.974459 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:14:31.974548 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:14:31.974931 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.967287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.98.60:38853)
0925 12:14:31.967339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.967341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.967348 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:14:31.967369 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.967370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.967384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.967472 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.967787 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.968492 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.969504 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.969757 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.969819 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.971884 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.971886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.971886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.971892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.973489 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.973875 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.973876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.973877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.973928 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.974119 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.974120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.974120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.974265 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.974324 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.974769 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.974842 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:14:31.976565 13795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: Bootstrap starting.
I20260925 12:14:31.977272 13795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:31.977579 13795 log.cc:824] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:31.980563 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.970706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.0.98.61:44341)
0925 12:14:31.970741 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.970742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.970961 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:14:31.971054 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.971055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.971064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.971091 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.971938 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.973194 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.974169 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.974463 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.974508 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.976416 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.976418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.976418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.976425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.978148 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.978823 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.978824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.978824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.978876 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.979206 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.979207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.979207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.979327 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.979414 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.980296 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.980395 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:14:31.982355 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.974459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.98.60:38853)
0925 12:14:31.974990 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.974992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.975042 (+ 50us) server_negotiation.cc:407] Connection header received
0925 12:14:31.975115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.975116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.975125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.975179 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.975899 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.976548 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.977510 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.977799 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.977842 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.979339 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.979340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.979341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.979347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.980903 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.981491 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.981492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.981492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.981555 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.981732 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.981733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.981733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.981881 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.981972 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.982150 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.982224 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":79,"threads_started":1}
I20260925 12:14:31.984198 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:31.974848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.0.98.62:39481)
0925 12:14:31.975041 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:31.975043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:31.975621 (+ 578us) server_negotiation.cc:407] Connection header received
0925 12:14:31.976450 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:31.976451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:31.976462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:31.976509 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:31.977207 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.977966 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.979031 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:31.979309 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:31.979349 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:31.981148 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:31.981149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:31.981150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:31.981156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:31.983123 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.983469 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.983469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.983470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.983528 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:31.983711 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:31.983712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:31.983713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:31.983819 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:31.983914 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:31.984032 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 12:14:31.984103 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 12:14:31.985070 13795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: No bootstrap required, opened a new log
I20260925 12:14:31.986510 13795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:31.986621 13795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:31.986642 13795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365fbfd53fea408fb21ac0f306fd8000, State: Initialized, Role: FOLLOWER
I20260925 12:14:31.986779 13795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:31.987099 13795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:31.987056 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:31.987172 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:14:31.988742 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:14:31.988793 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:14:31.989796 13727 master_runner.cc:186] Error getting master registration for 127.0.98.61:44341: OK, instance_id { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" instance_seqno: 1790338471766566 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:14:31.992710 13727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:14:32.000062 13870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } attempt: 3
Sep 25 12:14:32 dist-test-slave-2hqq krb5kdc[13714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:14:32.008307 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:32.000539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.60:38853)
0925 12:14:32.000585 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:32.000587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:32.000596 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:14:32.000623 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:32.000624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:32.000634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:32.000694 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:32.001061 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:32.001816 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:32.002842 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:32.003152 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:32.003226 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:32.005461 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:32.005463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:32.005463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:32.005470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:32.007104 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:32.007519 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:32.007520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:32.007520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:32.007579 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:32.007776 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:32.007776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:32.007777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:32.007911 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:32.007980 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:32.008127 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:14:32.008211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:14:32.008296 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:32.000326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38853 (local address 127.0.0.1:47842)
0925 12:14:32.000376 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:32.000387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:32.000424 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:32.000746 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:32.000748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:32.000932 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:32.000980 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:32.000983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:32.001922 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:32.001923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:32.002748 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:32.002750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:32.003235 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:32.003236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:32.003296 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:32.003781 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:32.003787 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:32.005355 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:32.007186 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:32.007187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:32.007188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:32.007425 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:32.007645 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:32.007646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:32.007646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:32.007713 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:32.007978 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:32.007979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:32.008025 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 12:14:32.008095 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 12:14:32.008191 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:14:32.010017 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: Bootstrap starting.
I20260925 12:14:32.010741 13870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:32.011037 13870 log.cc:824] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:32.012456 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: No bootstrap required, opened a new log
I20260925 12:14:32.014447 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:32.014582 13870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:32.014629 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0672b7d4205a4fc58b0f42bb13edf087, State: Initialized, Role: FOLLOWER
I20260925 12:14:32.014748 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:32.015044 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 12:14:32.016815 13998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:14:32.016923 13998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:14:32.015002 13987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.017750 13987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:32.019033 13802 master_runner.cc:263] No leader master found from master 0672b7d4205a4fc58b0f42bb13edf087
I20260925 12:14:32.019086 13802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:14:32.055791 13973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:32.055927 13973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:32.056160 13973 leader_election.cc:288] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0672b7d4205a4fc58b0f42bb13edf087 (127.0.98.61:44341), a6596e6e1c604ace9fca1be7291747a8 (127.0.98.60:38853)
I20260925 12:14:32.056584 13844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "365fbfd53fea408fb21ac0f306fd8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0672b7d4205a4fc58b0f42bb13edf087" is_pre_election: true
I20260925 12:14:32.056753 13844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365fbfd53fea408fb21ac0f306fd8000 in term 0.
I20260925 12:14:32.056941 13744 leader_election.cc:302] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0672b7d4205a4fc58b0f42bb13edf087, 365fbfd53fea408fb21ac0f306fd8000; no voters:
I20260925 12:14:32.057068 13973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:14:32.057121 13973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:14:32.057150 13973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:32.058035 13973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:32.058161 13973 leader_election.cc:288] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [CANDIDATE]: Term 1 election: Requested vote from peers 0672b7d4205a4fc58b0f42bb13edf087 (127.0.98.61:44341), a6596e6e1c604ace9fca1be7291747a8 (127.0.98.60:38853)
I20260925 12:14:32.058413 13844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "365fbfd53fea408fb21ac0f306fd8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0672b7d4205a4fc58b0f42bb13edf087"
I20260925 12:14:32.058497 13844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:32.059401 13844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365fbfd53fea408fb21ac0f306fd8000 in term 1.
I20260925 12:14:32.059556 13744 leader_election.cc:302] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0672b7d4205a4fc58b0f42bb13edf087, 365fbfd53fea408fb21ac0f306fd8000; no voters:
I20260925 12:14:32.059852 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "365fbfd53fea408fb21ac0f306fd8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6596e6e1c604ace9fca1be7291747a8" is_pre_election: true
I20260925 12:14:32.060020 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365fbfd53fea408fb21ac0f306fd8000 in term 0.
I20260925 12:14:32.060163 13918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "365fbfd53fea408fb21ac0f306fd8000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6596e6e1c604ace9fca1be7291747a8"
I20260925 12:14:32.060225 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:32.061055 13918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365fbfd53fea408fb21ac0f306fd8000 in term 1.
I20260925 12:14:32.061335 13973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:32.061509 13973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [term 1 LEADER]: Becoming Leader. State: Replica: 365fbfd53fea408fb21ac0f306fd8000, State: Running, Role: LEADER
I20260925 12:14:32.061614 13973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } }
I20260925 12:14:32.061990 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365fbfd53fea408fb21ac0f306fd8000. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.062080 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:32.062306 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:32.063429 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:32.065743 13918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [term 1 FOLLOWER]: Refusing update from remote peer 365fbfd53fea408fb21ac0f306fd8000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:32.066007 13844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [term 1 FOLLOWER]: Refusing update from remote peer 365fbfd53fea408fb21ac0f306fd8000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:32.066293 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:32.066640 13973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:32.067868 13987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365fbfd53fea408fb21ac0f306fd8000. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.067965 13987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:32.069025 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365fbfd53fea408fb21ac0f306fd8000. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.069160 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:32.071293 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.071395 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:32.071830 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.071915 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:32.072109 14003 mvcc.cc:205] Tried to move back new op lower bound from 7333226381578235904 to 7333226381565050880. Current Snapshot: MvccSnapshot[applied={T|T < 7333226381578235904}]
I20260925 12:14:32.072460 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.072556 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:32.072885 13987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "365fbfd53fea408fb21ac0f306fd8000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365fbfd53fea408fb21ac0f306fd8000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } peers { permanent_uuid: "0672b7d4205a4fc58b0f42bb13edf087" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44341 } } peers { permanent_uuid: "a6596e6e1c604ace9fca1be7291747a8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38853 } } }
I20260925 12:14:32.072960 13987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:14:32.073318 14006 mvcc.cc:205] Tried to move back new op lower bound from 7333226381578235904 to 7333226381565050880. Current Snapshot: MvccSnapshot[applied={T|T < 7333226381578235904}]
I20260925 12:14:32.073732 14004 mvcc.cc:205] Tried to move back new op lower bound from 7333226381578235904 to 7333226381565050880. Current Snapshot: MvccSnapshot[applied={T|T < 7333226381578235904}]
I20260925 12:14:32.074615 14002 catalog_manager.cc:1347] Generated new cluster ID: 17cf2e75951f4747964b3cb9f224401a
I20260925 12:14:32.074671 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:32.087314 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:14:32.088140 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:32.098434 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 365fbfd53fea408fb21ac0f306fd8000: Generated new TSK 0
I20260925 12:14:32.098630 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:14:32.970528 13962 catalog_manager.cc:1259] Loaded cluster ID: 17cf2e75951f4747964b3cb9f224401a
I20260925 12:14:32.970603 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: loading cluster ID for follower catalog manager: success
I20260925 12:14:32.971458 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: acquiring CA information for follower catalog manager: success
I20260925 12:14:32.971702 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6596e6e1c604ace9fca1be7291747a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:14:33.017395 13998 catalog_manager.cc:1259] Loaded cluster ID: 17cf2e75951f4747964b3cb9f224401a
I20260925 12:14:33.017475 13998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: loading cluster ID for follower catalog manager: success
I20260925 12:14:33.018349 13998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: acquiring CA information for follower catalog manager: success
I20260925 12:14:33.018684 13998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0672b7d4205a4fc58b0f42bb13edf087: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:14:33.159410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13727
I20260925 12:14:33.164901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13802
I20260925 12:14:33.169775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 13877
2026-09-25T12:14:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2830 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:14:33 dist-test-slave-2hqq krb5kdc[14021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:14:33 dist-test-slave-2hqq krb5kdc[14021](info): set up 2 sockets
Sep 25 12:14:33 dist-test-slave-2hqq krb5kdc[14021](info): commencing operation
krb5kdc: starting...
W20260925 12:14:34.204658 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:14:34.323021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:34.402794 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:34.402995 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:34.403017 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:34.404211 14034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:34.404258 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:34.404271 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:34.404311 14034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:34.404343 14034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:34.405892 14034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:34.406149 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:34.406392 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:34.411118 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:34.411104 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:34.411307 14034 server_base.cc:1034] running on GCE node
W20260925 12:14:34.411140 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:34.411659 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:34.411898 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:34.413031 14034 hybrid_clock.cc:648] HybridClock initialized: now 1790338474413017 us; error 29 us; skew 500 ppm
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:34.415370 14034 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:34.415815 14034 webserver.cc:460] Webserver started at http://127.0.98.62:38121/ using document root <none> and password file <none>
I20260925 12:14:34.416050 14034 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:34.416093 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:34.416249 14034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:34.417320 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "01d59ca1b8c14675a3d35adf0ef09d85"
format_stamp: "Formatted at 2026-09-25 12:14:34 on dist-test-slave-2hqq"
server_key: "f86b1325f7bf62b4feea1f682734d38f"
server_key_iv: "babe9dffa7500bed0af571c2c6e3a5e9"
server_key_version: "encryptionkey@0"
I20260925 12:14:34.417670 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "01d59ca1b8c14675a3d35adf0ef09d85"
format_stamp: "Formatted at 2026-09-25 12:14:34 on dist-test-slave-2hqq"
server_key: "f86b1325f7bf62b4feea1f682734d38f"
server_key_iv: "babe9dffa7500bed0af571c2c6e3a5e9"
server_key_version: "encryptionkey@0"
I20260925 12:14:34.418973 14034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:14:34.419646 14049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:34.419826 14034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:14:34.419921 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "01d59ca1b8c14675a3d35adf0ef09d85"
format_stamp: "Formatted at 2026-09-25 12:14:34 on dist-test-slave-2hqq"
server_key: "f86b1325f7bf62b4feea1f682734d38f"
server_key_iv: "babe9dffa7500bed0af571c2c6e3a5e9"
server_key_version: "encryptionkey@0"
I20260925 12:14:34.420001 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:34.437845 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:34.439733 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:34.439901 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:34.444047 14034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33825
I20260925 12:14:34.444093 14101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33825 every 8 connection(s)
I20260925 12:14:34.444459 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:34.445349 14102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:34.447499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14034
I20260925 12:14:34.447613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:14:34.447616 14102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85: Bootstrap starting.
I20260925 12:14:34.447846 392 external_mini_cluster.cc:1428] Setting key d241390fdd95489ed4c035420d1ef9a5
I20260925 12:14:34.448252 14102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:34.448644 14102 log.cc:824] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:34.449378 14102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85: No bootstrap required, opened a new log
I20260925 12:14:34.451649 14102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260925 12:14:34.451818 14102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:34.451845 14102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d59ca1b8c14675a3d35adf0ef09d85, State: Initialized, Role: FOLLOWER
I20260925 12:14:34.451984 14102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260925 12:14:34.452056 14102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:34.452078 14102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:34.452139 14102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:34.452785 14102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260925 12:14:34.452888 14102 leader_election.cc:302] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01d59ca1b8c14675a3d35adf0ef09d85; no voters:
I20260925 12:14:34.453038 14102 leader_election.cc:288] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:14:34.453096 14107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:34.453289 14102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:34.453305 14107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [term 1 LEADER]: Becoming Leader. State: Replica: 01d59ca1b8c14675a3d35adf0ef09d85, State: Running, Role: LEADER
I20260925 12:14:34.453413 14107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260925 12:14:34.453723 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d59ca1b8c14675a3d35adf0ef09d85. Latest consensus state: current_term: 1 leader_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } } }
I20260925 12:14:34.453787 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:34.454437 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d59ca1b8c14675a3d35adf0ef09d85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } } }
I20260925 12:14:34.454523 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:34.454655 14115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:34.455138 14115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:34.456843 14115 catalog_manager.cc:1347] Generated new cluster ID: ed253495cc414359bf3c5c8b79b6add6
I20260925 12:14:34.456892 14115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:34.463761 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:34.448352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.98.62:33825)
0925 12:14:34.448579 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:34.448589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:34.448626 (+ 37us) server_negotiation.cc:407] Connection header received
0925 12:14:34.449227 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:34.449235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:34.449295 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:34.449451 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:34.450138 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:34.451617 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:34.452587 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:34.453230 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:34.453690 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:34.456462 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:34.456466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:34.456467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:34.456484 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:34.462133 (+ 5649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:34.462532 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:34.462536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:34.462536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:34.462661 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:34.462858 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:34.462859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:34.462859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:34.463175 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:34.463334 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:34.463408 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 12:14:34.463542 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260925 12:14:34.464720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:14:34.466104 14115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:14:34.466574 14115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:34.470007 14115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d59ca1b8c14675a3d35adf0ef09d85: Generated new TSK 0
I20260925 12:14:34.470173 14115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:14:34.518770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14034
2026-09-25T12:14:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1347 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14129](info): set up 2 sockets
Sep 25 12:14:34 dist-test-slave-2hqq krb5kdc[14129](info): commencing operation
krb5kdc: starting...
W20260925 12:14:35.580561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:14:35 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:14:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:14:35.703536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:35.784565 14142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:35.784781 14142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:35.784803 14142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:35.785913 14142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:35.785948 14142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:35.785964 14142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:35.785975 14142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:35.785987 14142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:35.787381 14142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:35.787613 14142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:35.787786 14142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:14:35.792794 14142 server_base.cc:1034] running on GCE node
W20260925 12:14:35.792776 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:35.792762 14147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:35.792776 14148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:35.793270 14142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:35.793607 14142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:35.794808 14142 hybrid_clock.cc:648] HybridClock initialized: now 1790338475794785 us; error 34 us; skew 500 ppm
Sep 25 12:14:35 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:35.797629 14142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:35.798166 14142 webserver.cc:460] Webserver started at http://127.0.98.62:33047/ using document root <none> and password file <none>
I20260925 12:14:35.798408 14142 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:35.798486 14142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:35.798640 14142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:35.799782 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "ccc95361f1af4e758e08486b6750ab40"
format_stamp: "Formatted at 2026-09-25 12:14:35 on dist-test-slave-2hqq"
server_key: "547cecf2859ac4a7b4becbd1d1cda543"
server_key_iv: "26749bbe258e2b5a2238631ba3c02122"
server_key_version: "encryptionkey@0"
I20260925 12:14:35.800164 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccc95361f1af4e758e08486b6750ab40"
format_stamp: "Formatted at 2026-09-25 12:14:35 on dist-test-slave-2hqq"
server_key: "547cecf2859ac4a7b4becbd1d1cda543"
server_key_iv: "26749bbe258e2b5a2238631ba3c02122"
server_key_version: "encryptionkey@0"
I20260925 12:14:35.801704 14142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:14:35.802531 14157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:35.802726 14142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:35.802829 14142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccc95361f1af4e758e08486b6750ab40"
format_stamp: "Formatted at 2026-09-25 12:14:35 on dist-test-slave-2hqq"
server_key: "547cecf2859ac4a7b4becbd1d1cda543"
server_key_iv: "26749bbe258e2b5a2238631ba3c02122"
server_key_version: "encryptionkey@0"
I20260925 12:14:35.802899 14142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:35.815721 14142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:35.817796 14142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:35.817951 14142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:35.821828 14142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35973
I20260925 12:14:35.821866 14209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35973 every 8 connection(s)
I20260925 12:14:35.822219 14142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:35.823238 14210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:35.825639 14210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap starting.
I20260925 12:14:35.826192 14210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:35.826426 14210 log.cc:824] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:35.827185 14210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: No bootstrap required, opened a new log
I20260925 12:14:35.828402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14142
I20260925 12:14:35.828531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:14:35.828723 392 external_mini_cluster.cc:1428] Setting key 7e56c6d8afb0ee8d9e94e1fbfbe78f69
I20260925 12:14:35.828998 14210 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:35.829192 14210 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:35.829233 14210 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Initialized, Role: FOLLOWER
I20260925 12:14:35.829381 14210 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:35.829502 14210 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:35.829545 14210 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:35.829600 14210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:35.830803 14210 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:35.830924 14210 leader_election.cc:302] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc95361f1af4e758e08486b6750ab40; no voters:
I20260925 12:14:35.831094 14210 leader_election.cc:288] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:14:35.831137 14215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:35.831271 14215 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 LEADER]: Becoming Leader. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Running, Role: LEADER
I20260925 12:14:35.831357 14210 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:35.831354 14215 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:35.831892 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc95361f1af4e758e08486b6750ab40. Latest consensus state: current_term: 1 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:35.831959 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:35.832082 14216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:35.832139 14216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:35.832631 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:35.833143 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:35.834692 14220 catalog_manager.cc:1347] Generated new cluster ID: a499f14429c44fa8a1a43a7a556e110a
I20260925 12:14:35.834743 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:14:35 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:35.840615 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:35.829254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.98.62:35973)
0925 12:14:35.829433 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:35.829441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:35.829469 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:14:35.829591 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:35.829599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:35.829659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:35.829785 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:35.830614 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:35.832068 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:35.833246 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:35.833919 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:35.834024 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:35.836863 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:35.836866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:35.836867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:35.836879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:35.838802 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:35.839234 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:35.839237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:35.839237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:35.839346 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:35.839567 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:35.839568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:35.839569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:35.839867 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:35.840043 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:35.840138 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 12:14:35.840368 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260925 12:14:35.868820 14220 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:14:35.869596 14220 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:35.877250 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Generated new TSK 0
I20260925 12:14:35.877475 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:14:35.905798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:35.996562 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:35.996773 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:35.996797 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:35.997920 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:35.997958 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:35.997984 14236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:35.999354 14236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:35.999560 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:35.999739 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:36.004949 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.004949 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.004968 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.005266 14236 server_base.cc:1034] running on GCE node
I20260925 12:14:36.005481 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.005734 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.006866 14236 hybrid_clock.cc:648] HybridClock initialized: now 1790338476006855 us; error 28 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.009327 14236 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:36.009840 14236 webserver.cc:460] Webserver started at http://127.0.98.1:45945/ using document root <none> and password file <none>
I20260925 12:14:36.010073 14236 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.010118 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.010274 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:36.011307 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "dd85d03a5e74959bd473186a51245bac"
server_key_iv: "e5ce69b64308ed87885b4bde4a88ebac"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.011657 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "dd85d03a5e74959bd473186a51245bac"
server_key_iv: "e5ce69b64308ed87885b4bde4a88ebac"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.013085 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:14:36.013789 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.013962 14236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:14:36.014048 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "dd85d03a5e74959bd473186a51245bac"
server_key_iv: "e5ce69b64308ed87885b4bde4a88ebac"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.014130 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.025110 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.027033 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.027194 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.027416 14236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:36.027714 14236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:36.027762 14236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.027796 14236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:36.027822 14236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.034215 14236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260925 12:14:36.034291 14364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260925 12:14:36.034605 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:36.040978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14236
I20260925 12:14:36.041087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:14:36.041306 392 external_mini_cluster.cc:1428] Setting key f7affa10745ebfb1fe5932407b0e7186
I20260925 12:14:36.044800 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.035320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33713 (local address 127.0.98.62:35973)
0925 12:14:36.035382 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.035384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.035574 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:14:36.036102 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.036103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.036117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.036176 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.037205 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.037704 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.038696 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.038883 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.038935 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.041285 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.041287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.041287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.041294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.043264 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.043742 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.043743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.043743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.043805 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.044089 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.044090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.044090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.044207 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.044282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.044597 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.044703 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:14:36.045068 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.035277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:33713)
0925 12:14:36.035431 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.035474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.036012 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.036228 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.036231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.036924 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.037123 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.037128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.037785 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.037786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.038619 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.038622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.038952 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.038953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.039071 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.039557 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.039565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.041187 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.043363 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.043370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.043372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.043661 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.043874 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.043874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.043875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.043998 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.044273 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.044280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.044434 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.044573 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.044756 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 12:14:36.045547 14365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:36.045644 14365 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:36.045845 14365 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:36.046378 14174 ts_manager.cc:194] Registered new tserver with Master: d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269)
I20260925 12:14:36.046988 14174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33713
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:14:36.068472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:36.150345 14370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:36.150566 14370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:36.150591 14370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:36.151810 14370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:36.151852 14370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:36.151880 14370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:36.153366 14370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:36.153569 14370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:36.153745 14370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:36.159307 14378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.159338 14370 server_base.cc:1034] running on GCE node
W20260925 12:14:36.159307 14375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.159309 14376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.159860 14370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.160079 14370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.161212 14370 hybrid_clock.cc:648] HybridClock initialized: now 1790338476161201 us; error 27 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.163736 14370 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:36.164199 14370 webserver.cc:460] Webserver started at http://127.0.98.2:45227/ using document root <none> and password file <none>
I20260925 12:14:36.164433 14370 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.164477 14370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.164676 14370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:36.165732 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "86eb475ee97bd5b2406edbce01c64094"
server_key_iv: "a7decc6c693681f6a47461cb1cd02ce2"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.166090 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "86eb475ee97bd5b2406edbce01c64094"
server_key_iv: "a7decc6c693681f6a47461cb1cd02ce2"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.167372 14370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:14:36.168108 14385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.168270 14370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:14:36.168363 14370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "86eb475ee97bd5b2406edbce01c64094"
server_key_iv: "a7decc6c693681f6a47461cb1cd02ce2"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.168438 14370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.198868 14370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.201154 14370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.201418 14370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.201668 14370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:36.201979 14370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:36.202031 14370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.202065 14370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:36.202092 14370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.208011 14370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34427
I20260925 12:14:36.208062 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34427 every 8 connection(s)
I20260925 12:14:36.208385 14370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:14:36.213697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14370
I20260925 12:14:36.213810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:14:36.213976 392 external_mini_cluster.cc:1428] Setting key acc16d74c351ff986a44f1e42bec6abe
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:36.219385 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.209123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48691 (local address 127.0.98.62:35973)
0925 12:14:36.209178 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.209180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.209365 (+ 185us) server_negotiation.cc:407] Connection header received
0925 12:14:36.209973 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.209975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.209990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.210066 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.211005 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.211528 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.212818 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.213029 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.213098 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.216204 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.216205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.216205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.216212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.217732 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.218307 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.218308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.218309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.218359 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.218682 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.218683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.218684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.218809 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.218864 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.219244 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.219319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:14:36.219707 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.209110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:48691)
0925 12:14:36.209255 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.209290 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.209800 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.210142 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.210146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.210724 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.210915 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.210920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.211605 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.211606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.212415 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.212417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.213133 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.213134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.213255 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.213965 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.213973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.216098 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.217817 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.217824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.217826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.218228 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.218425 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.218426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.218426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.218596 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.218882 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.218889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.219090 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.219217 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.219409 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 12:14:36.220271 14499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:36.220358 14499 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:36.220590 14499 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:36.220983 14174 ts_manager.cc:194] Registered new tserver with Master: 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427)
I20260925 12:14:36.221364 14174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48691
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:14:36.243558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:36.324200 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:36.324455 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:36.324492 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:36.325685 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:36.325755 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:36.325800 14504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:36.327289 14504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:36.327556 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:36.327785 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:36.332842 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.332927 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.332967 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.333082 14504 server_base.cc:1034] running on GCE node
I20260925 12:14:36.333297 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.333544 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.334707 14504 hybrid_clock.cc:648] HybridClock initialized: now 1790338476334691 us; error 31 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.337177 14504 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:36.337595 14504 webserver.cc:460] Webserver started at http://127.0.98.3:36481/ using document root <none> and password file <none>
I20260925 12:14:36.337857 14504 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.337929 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.338079 14504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:36.339121 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "720b5fc494def03cdf54c2f714ba301b"
server_key_iv: "7c32c68b8896bb230ae2da96fe298ef8"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.339478 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "720b5fc494def03cdf54c2f714ba301b"
server_key_iv: "7c32c68b8896bb230ae2da96fe298ef8"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.340931 14504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 12:14:36.341645 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.341797 14504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:14:36.341889 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "720b5fc494def03cdf54c2f714ba301b"
server_key_iv: "7c32c68b8896bb230ae2da96fe298ef8"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.341985 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.356752 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.358646 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.358809 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.359037 14504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:36.359337 14504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:36.359398 14504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.359440 14504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:36.359467 14504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.365370 14504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33333
I20260925 12:14:36.365406 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33333 every 8 connection(s)
I20260925 12:14:36.365724 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:14:36.368363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14504
I20260925 12:14:36.368479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:14:36.368670 392 external_mini_cluster.cc:1428] Setting key 582175eebef4da16f57ee8dd3e901a31
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:36.375162 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.366347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34173 (local address 127.0.98.62:35973)
0925 12:14:36.366397 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.366399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.366607 (+ 208us) server_negotiation.cc:407] Connection header received
0925 12:14:36.367153 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.367154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.367168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.367221 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.368207 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.368858 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.370126 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.370333 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.370421 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.372710 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.372711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.372712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.372718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.373969 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.374320 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.374321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.374321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.374370 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.374633 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.374634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.374634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.374732 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.374780 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.375029 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.375097 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:14:36.375298 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.366322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:34173)
0925 12:14:36.366470 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.366512 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.367034 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.367296 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.367301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.367884 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.368082 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.368087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.369022 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.369022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.370027 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.370030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.370429 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.370430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.370563 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.371100 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.371108 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.372453 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.374040 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.374046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.374047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.374260 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.374480 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.374481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.374481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.374581 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.374780 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.374784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.374894 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.374974 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.375092 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260925 12:14:36.375737 14633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:36.375838 14633 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:36.376019 14633 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:36.376379 14174 ts_manager.cc:194] Registered new tserver with Master: 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
I20260925 12:14:36.376765 14174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34173
I20260925 12:14:36.380721 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:14:36.387717 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.382068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.62:35973)
0925 12:14:36.382127 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.382129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.382170 (+ 41us) server_negotiation.cc:407] Connection header received
0925 12:14:36.382237 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.382237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.382246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.382303 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.382815 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.383290 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.383991 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.384173 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.384228 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.385593 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.385594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.385594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.385600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.386787 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.387083 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.387084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.387084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.387132 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.387285 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.387285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.387285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.387418 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.387503 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.387580 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.387652 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:14:36.389503 14174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:14:36.391160 14174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:14:36.397850 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.393566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:47776)
0925 12:14:36.393665 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.393705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.393765 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.394095 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.394099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.394101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:36.394174 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.394176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395445 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.395446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.396690 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.396693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397521 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.397597 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.397611 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.397690 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.397747 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":55,"threads_started":1}
I20260925 12:14:36.397857 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.393679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.0.98.1:37269)
0925 12:14:36.393896 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.393901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.393910 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:14:36.393967 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.393986 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.394004 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.394091 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:36.394331 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395329 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.396770 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397436 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397520 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.397579 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.397711 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.397752 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":109,"threads_started":1}
I20260925 12:14:36.398161 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.393946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34427 (local address 127.0.0.1:53764)
0925 12:14:36.394262 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.394273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.394322 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.394892 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.394893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.394894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:36.394938 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.394939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395977 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.395978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.397072 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.397073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397914 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.397984 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.397988 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.398059 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.398101 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":62,"threads_started":1}
I20260925 12:14:36.398228 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.394077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.0.98.2:34427)
0925 12:14:36.394748 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.394752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.394760 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:14:36.394794 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.394796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.394810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.394894 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:36.394998 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395862 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397163 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397759 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397826 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.397878 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.398089 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.398129 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":99,"threads_started":1}
I20260925 12:14:36.398303 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.393778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33333 (local address 127.0.0.1:33584)
0925 12:14:36.393889 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.393898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.393943 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.394613 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.394614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.394617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:36.394675 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.394677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395809 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.395810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.396993 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.396996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397977 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.398067 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.398071 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.398123 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.398168 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":77,"threads_started":1}
I20260925 12:14:36.398331 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.394078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.0.98.3:33333)
0925 12:14:36.394461 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.394465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.394475 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:14:36.394514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.394515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.394527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.394602 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:36.394757 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.395707 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397076 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.397794 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.397871 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.397936 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.398211 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.398250 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":93,"threads_started":1}
I20260925 12:14:36.399214 14433 tablet_service.cc:1467] Processing CreateTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 (DEFAULT_TABLE table=test-table [id=99edacb2b4a8498280eb24a2b655cbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:36.399204 14567 tablet_service.cc:1467] Processing CreateTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 (DEFAULT_TABLE table=test-table [id=99edacb2b4a8498280eb24a2b655cbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:36.399525 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493d11bbb3144ca1a466ada6fe5df2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:36.399555 14567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493d11bbb3144ca1a466ada6fe5df2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:36.399616 14299 tablet_service.cc:1467] Processing CreateTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 (DEFAULT_TABLE table=test-table [id=99edacb2b4a8498280eb24a2b655cbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:36.399888 14299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493d11bbb3144ca1a466ada6fe5df2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:36.402161 14650 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Bootstrap starting.
I20260925 12:14:36.402782 14650 tablet_bootstrap.cc:654] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:36.403206 14651 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Bootstrap starting.
I20260925 12:14:36.403422 14650 log.cc:824] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:36.403780 14652 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap starting.
I20260925 12:14:36.404103 14651 tablet_bootstrap.cc:654] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:36.404368 14652 tablet_bootstrap.cc:654] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:36.404604 14650 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: No bootstrap required, opened a new log
I20260925 12:14:36.404690 14650 ts_tablet_manager.cc:1397] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:14:36.404703 14651 log.cc:824] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:36.404973 14652 log.cc:824] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:36.405468 14651 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: No bootstrap required, opened a new log
I20260925 12:14:36.405563 14651 ts_tablet_manager.cc:1397] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 12:14:36.405718 14652 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: No bootstrap required, opened a new log
I20260925 12:14:36.405794 14652 ts_tablet_manager.cc:1397] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:14:36.406765 14650 raft_consensus.cc:348] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.406952 14650 raft_consensus.cc:374] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:36.406991 14650 raft_consensus.cc:729] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998f8a1a6f264f2388c0711ecb8bb0fd, State: Initialized, Role: FOLLOWER
I20260925 12:14:36.407120 14650 consensus_queue.cc:260] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.407405 14650 ts_tablet_manager.cc:1428] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:14:36.407538 14499 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
I20260925 12:14:36.407691 14652 raft_consensus.cc:348] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.407864 14652 raft_consensus.cc:374] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:36.407909 14652 raft_consensus.cc:729] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b2d474d05143a3a07d1b5b8faf085b, State: Initialized, Role: FOLLOWER
I20260925 12:14:36.407889 14651 raft_consensus.cc:348] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.408066 14651 raft_consensus.cc:374] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:36.408023 14652 consensus_queue.cc:260] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.408110 14651 raft_consensus.cc:729] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365281ddf2d14ea9b855fc7dda1e9c5e, State: Initialized, Role: FOLLOWER
I20260925 12:14:36.408212 14651 consensus_queue.cc:260] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.408303 14652 ts_tablet_manager.cc:1428] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:14:36.408414 14365 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
I20260925 12:14:36.408639 14651 ts_tablet_manager.cc:1428] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:14:36.408749 14633 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
I20260925 12:14:36.455154 14656 raft_consensus.cc:482] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:36.455279 14656 raft_consensus.cc:504] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.455700 14656 leader_election.cc:288] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:36.458928 14500 tablet.cc:2369] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:14:36.459677 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.455895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.98.2:56391)
0925 12:14:36.455953 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.455981 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.456050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.456226 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.456227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.456229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:36.456314 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.456316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.457278 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.457279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.458632 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.458635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.459381 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.459382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.459504 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.459507 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.459560 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.459588 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":61}
I20260925 12:14:36.459688 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.455997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56391 (local address 127.0.98.1:37269)
0925 12:14:36.456087 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.456089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.456098 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:14:36.456128 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.456129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.456142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.456205 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:36.456415 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.457187 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.458728 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.459296 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.459371 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.459382 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.459579 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.459618 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 12:14:36.459688 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.456088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52729 (local address 127.0.98.3:33333)
0925 12:14:36.456134 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.456136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.456244 (+ 108us) server_negotiation.cc:407] Connection header received
0925 12:14:36.456277 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.456279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.456295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.456337 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:36.456549 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.457224 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.458728 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.459296 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.459377 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.459387 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.459579 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.459618 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:14:36.459685 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.456000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33333 (local address 127.0.98.2:52729)
0925 12:14:36.456169 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.456177 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.456226 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.456367 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.456368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.456369 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:36.456433 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.456434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.457289 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.457290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.458632 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.458635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.459383 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.459384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.459467 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.459472 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.459537 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.459583 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":90,"thread_start_us":120,"threads_started":1}
I20260925 12:14:36.460106 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493d11bbb3144ca1a466ada6fe5df2a0" candidate_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" is_pre_election: true
I20260925 12:14:36.460139 14319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493d11bbb3144ca1a466ada6fe5df2a0" candidate_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" is_pre_election: true
I20260925 12:14:36.460258 14587 raft_consensus.cc:2454] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998f8a1a6f264f2388c0711ecb8bb0fd in term 0.
I20260925 12:14:36.460300 14319 raft_consensus.cc:2454] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998f8a1a6f264f2388c0711ecb8bb0fd in term 0.
I20260925 12:14:36.460585 14389 leader_election.cc:302] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998f8a1a6f264f2388c0711ecb8bb0fd, d9b2d474d05143a3a07d1b5b8faf085b; no voters:
I20260925 12:14:36.460714 14656 raft_consensus.cc:2781] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:14:36.460793 14656 raft_consensus.cc:482] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:14:36.460826 14656 raft_consensus.cc:3037] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:36.461549 14656 raft_consensus.cc:504] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.461689 14656 leader_election.cc:288] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [CANDIDATE]: Term 1 election: Requested vote from peers d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
I20260925 12:14:36.461946 14319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493d11bbb3144ca1a466ada6fe5df2a0" candidate_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
I20260925 12:14:36.461978 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493d11bbb3144ca1a466ada6fe5df2a0" candidate_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
I20260925 12:14:36.462044 14319 raft_consensus.cc:3037] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:36.462059 14587 raft_consensus.cc:3037] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:36.463042 14587 raft_consensus.cc:2454] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998f8a1a6f264f2388c0711ecb8bb0fd in term 1.
I20260925 12:14:36.463061 14319 raft_consensus.cc:2454] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998f8a1a6f264f2388c0711ecb8bb0fd in term 1.
I20260925 12:14:36.463246 14389 leader_election.cc:302] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd; no voters:
I20260925 12:14:36.463353 14656 raft_consensus.cc:2781] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:36.463586 14656 raft_consensus.cc:686] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 LEADER]: Becoming Leader. State: Replica: 998f8a1a6f264f2388c0711ecb8bb0fd, State: Running, Role: LEADER
I20260925 12:14:36.463685 14656 consensus_queue.cc:237] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.464522 14174 catalog_manager.cc:5495] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } health_report { overall_health: HEALTHY } } }
I20260925 12:14:36.472748 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.470357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.98.2:34427)
0925 12:14:36.470461 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.470463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.470471 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:14:36.470518 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.470519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.470543 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.470616 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:14:36.470748 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.471223 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.471976 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.472185 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.472236 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.472362 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:14:36.472559 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:14:36.472637 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.472655 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.472673 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 12:14:36.474516 14587 raft_consensus.cc:1261] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Refusing update from remote peer 998f8a1a6f264f2388c0711ecb8bb0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:36.474557 14319 raft_consensus.cc:1261] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Refusing update from remote peer 998f8a1a6f264f2388c0711ecb8bb0fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:36.474907 14656 consensus_queue.cc:1035] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:36.475049 14660 consensus_queue.cc:1035] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:36.477635 14665 mvcc.cc:205] Tried to move back new op lower bound from 7333226399637082112 to 7333226399595954176. Current Snapshot: MvccSnapshot[applied={T|T < 7333226399637082112}]
I20260925 12:14:36.477898 14664 mvcc.cc:205] Tried to move back new op lower bound from 7333226399637082112 to 7333226399595954176. Current Snapshot: MvccSnapshot[applied={T|T < 7333226399637082112}]
I20260925 12:14:36.479815 14172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:36.479929 14172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:36.480700 14172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Sending DeleteTablet for 3 replicas of tablet 493d11bbb3144ca1a466ada6fe5df2a0
I20260925 12:14:36.481065 14299 tablet_service.cc:1514] Processing DeleteTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47776
I20260925 12:14:36.481123 14567 tablet_service.cc:1514] Processing DeleteTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33584
I20260925 12:14:36.481110 14433 tablet_service.cc:1514] Processing DeleteTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53764
I20260925 12:14:36.481345 14675 tablet_replica.cc:330] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: stopping tablet replica
I20260925 12:14:36.481349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14236
I20260925 12:14:36.481447 14675 raft_consensus.cc:2229] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:14:36.481568 14675 raft_consensus.cc:2258] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:36.482713 14675 ts_tablet_manager.cc:1905] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:36.484623 14675 ts_tablet_manager.cc:1918] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:36.484699 14675 log.cc:1196] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/493d11bbb3144ca1a466ada6fe5df2a0
I20260925 12:14:36.484979 14675 ts_tablet_manager.cc:1939] T 493d11bbb3144ca1a466ada6fe5df2a0 P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting consensus metadata
I20260925 12:14:36.485412 14161 catalog_manager.cc:4841] TS 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): tablet 493d11bbb3144ca1a466ada6fe5df2a0 (table test-table [id=99edacb2b4a8498280eb24a2b655cbda]) successfully deleted
W20260925 12:14:36.486706 14161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37269 (error 108)
I20260925 12:14:36.486936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14370
I20260925 12:14:36.487550 14674 tablet_replica.cc:330] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: stopping tablet replica
I20260925 12:14:36.487676 14674 raft_consensus.cc:2229] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:36.487788 14674 raft_consensus.cc:2258] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:36.487911 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.487104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:47790)
0925 12:14:36.487803 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.487845 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677}
I20260925 12:14:36.488250 14674 ts_tablet_manager.cc:1905] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 12:14:36.489035 14161 catalog_manager.cc:4584] TS d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 493d11bbb3144ca1a466ada6fe5df2a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
I20260925 12:14:36.489810 14674 ts_tablet_manager.cc:1918] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:36.489862 14674 log.cc:1196] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/493d11bbb3144ca1a466ada6fe5df2a0
I20260925 12:14:36.490079 14674 ts_tablet_manager.cc:1939] T 493d11bbb3144ca1a466ada6fe5df2a0 P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting consensus metadata
I20260925 12:14:36.490449 14161 catalog_manager.cc:4841] TS 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): tablet 493d11bbb3144ca1a466ada6fe5df2a0 (table test-table [id=99edacb2b4a8498280eb24a2b655cbda]) successfully deleted
I20260925 12:14:36.492481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14504
I20260925 12:14:36.497736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14142
I20260925 12:14:36.503295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--webserver_interface=127.0.98.62
--webserver_port=33047
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:36.584260 14677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:36.584537 14677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:36.584589 14677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:36.585772 14677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:36.585835 14677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:36.585860 14677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:36.585887 14677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:36.585916 14677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:36.587394 14677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:36.587671 14677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:36.587906 14677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:36.596896 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.596905 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.596917 14686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.597188 14677 server_base.cc:1034] running on GCE node
I20260925 12:14:36.597456 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.597779 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.598961 14677 hybrid_clock.cc:648] HybridClock initialized: now 1790338476598961 us; error 59 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.607125 14677 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:36.607609 14677 webserver.cc:460] Webserver started at http://127.0.98.62:33047/ using document root <none> and password file <none>
I20260925 12:14:36.607890 14677 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.607956 14677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.609753 14677 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:36.610580 14693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.610814 14677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:36.610916 14677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccc95361f1af4e758e08486b6750ab40"
format_stamp: "Formatted at 2026-09-25 12:14:35 on dist-test-slave-2hqq"
server_key: "547cecf2859ac4a7b4becbd1d1cda543"
server_key_iv: "26749bbe258e2b5a2238631ba3c02122"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.611362 14677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.627596 14677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.630158 14677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.630329 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.634418 14677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35973
I20260925 12:14:36.634451 14745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35973 every 8 connection(s)
I20260925 12:14:36.634855 14677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:36.636682 14746 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:14:36.637348 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap starting.
I20260925 12:14:36.638592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14677
I20260925 12:14:36.638865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37269
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=45945
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:36.645797 14746 log.cc:824] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:36.647761 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:14:36.648053 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap complete.
I20260925 12:14:36.650383 14746 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:36.650633 14746 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Initialized, Role: FOLLOWER
I20260925 12:14:36.650830 14746 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:36.650918 14746 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:36.650982 14746 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:36.651032 14746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:14:36.652037 14746 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:36.652156 14746 leader_election.cc:302] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc95361f1af4e758e08486b6750ab40; no voters:
I20260925 12:14:36.652352 14746 leader_election.cc:288] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:14:36.652391 14750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:14:36.652539 14750 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 LEADER]: Becoming Leader. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Running, Role: LEADER
I20260925 12:14:36.652637 14746 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:36.652626 14750 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:36.652918 14750 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc95361f1af4e758e08486b6750ab40. Latest consensus state: current_term: 2 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:36.652992 14750 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:36.652927 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:36.653297 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:36.653209 14754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:36.654222 14754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:36.654371 14754 catalog_manager.cc:1259] Loaded cluster ID: a499f14429c44fa8a1a43a7a556e110a
I20260925 12:14:36.654435 14754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:36.654850 14754 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 12:14:36.659785 14754 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 12:14:36.662323 14754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:36.662539 14754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Loaded TSK: 0
I20260925 12:14:36.662748 14754 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:14:36.724706 14748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:36.724920 14748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:36.724943 14748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:36.726140 14748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:36.726187 14748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:36.726217 14748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:36.727691 14748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45945
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:36.727924 14748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:36.728183 14748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:36.733238 14773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.733230 14774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.733449 14748 server_base.cc:1034] running on GCE node
W20260925 12:14:36.733245 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.733734 14748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.734010 14748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.735163 14748 hybrid_clock.cc:648] HybridClock initialized: now 1790338476735153 us; error 28 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.737875 14748 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:36.738445 14748 webserver.cc:460] Webserver started at http://127.0.98.1:45945/ using document root <none> and password file <none>
I20260925 12:14:36.738709 14748 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.738796 14748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.740674 14748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:14:36.741463 14783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.741698 14748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:36.741791 14748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "dd85d03a5e74959bd473186a51245bac"
server_key_iv: "e5ce69b64308ed87885b4bde4a88ebac"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.742242 14748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.767539 14748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.770123 14748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.770279 14748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.770547 14748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:36.771075 14790 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:14:36.772535 14748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:14:36.772598 14748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:14:36.772634 14748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:14:36.773485 14748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:14:36.773530 14748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:36.773618 14790 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap starting.
I20260925 12:14:36.780049 14748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260925 12:14:36.780311 14897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260925 12:14:36.780454 14748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:14:36.784132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14748
I20260925 12:14:36.784430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34427
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=45227
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:36.784698 14790 log.cc:824] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:36.787765 14790 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:14:36.788089 14790 tablet_bootstrap.cc:492] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap complete.
I20260925 12:14:36.789300 14790 ts_tablet_manager.cc:1397] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260925 12:14:36.790720 14790 raft_consensus.cc:348] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.791088 14790 raft_consensus.cc:729] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b2d474d05143a3a07d1b5b8faf085b, State: Initialized, Role: FOLLOWER
I20260925 12:14:36.791226 14790 consensus_queue.cc:260] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } }
I20260925 12:14:36.791508 14790 ts_tablet_manager.cc:1428] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:36.797147 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.781222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:36633)
0925 12:14:36.781429 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.781477 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.782125 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.785742 (+ 3617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.785747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.786116 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.786410 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.786420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.788009 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.788010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.789289 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.789292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.790083 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.790084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.790230 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.790840 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.790851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.792854 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.795343 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.795351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.795353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.795699 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.796015 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.796016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.796016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.796125 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.796658 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.796663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.796780 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.796869 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.797003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260925 12:14:36.797353 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.781335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36633 (local address 127.0.98.62:35973)
0925 12:14:36.785265 (+ 3930us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.785276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.785322 (+ 46us) server_negotiation.cc:407] Connection header received
0925 12:14:36.785459 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.785466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.785527 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.785683 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.786563 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.787893 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.789393 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.789979 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.790075 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.792983 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.792988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.792988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.793003 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.795239 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.795799 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.795803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.795803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.795928 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.796200 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.796201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.796201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.796580 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.796677 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.796955 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.797117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3871,"thread_start_us":90,"threads_started":1}
I20260925 12:14:36.797986 14898 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:36.798097 14898 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:36.798274 14898 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:36.798985 14710 ts_manager.cc:194] Registered new tserver with Master: d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269)
I20260925 12:14:36.800130 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36633
I20260925 12:14:36.801036 14898 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 12:14:36.807799 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.799551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:47796)
0925 12:14:36.799718 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.799760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.799830 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.800035 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.800041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.800416 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.800549 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.800552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.801637 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.801638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.802565 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.802568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.802975 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.802979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.803053 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.803490 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.803495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.804960 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.806678 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.806684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.806686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.806885 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.807094 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.807095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.807095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.807154 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.807425 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.807438 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.807490 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.807585 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.807701 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 12:14:36.807796 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.799643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.98.1:37269)
0925 12:14:36.799793 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.799798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.799808 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:14:36.799925 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.799934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.799949 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.800019 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.800631 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.801568 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.802646 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.802914 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.802964 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.805063 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.805070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.805071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.805078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.806576 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.806968 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.806971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.806972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.807029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.807220 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.807221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.807221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.807357 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.807446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.807620 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.807708 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 12:14:36.808197 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet 493d11bbb3144ca1a466ada6fe5df2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47796
I20260925 12:14:36.808772 14907 tablet_replica.cc:330] stopping tablet replica
I20260925 12:14:36.808887 14907 raft_consensus.cc:2229] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:36.808952 14907 raft_consensus.cc:2258] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:36.809257 14907 ts_tablet_manager.cc:1905] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:36.811676 14907 ts_tablet_manager.cc:1918] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:36.811758 14907 log.cc:1196] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/493d11bbb3144ca1a466ada6fe5df2a0
I20260925 12:14:36.811990 14907 ts_tablet_manager.cc:1939] T 493d11bbb3144ca1a466ada6fe5df2a0 P d9b2d474d05143a3a07d1b5b8faf085b: Deleting consensus metadata
I20260925 12:14:36.812520 14697 catalog_manager.cc:4841] TS d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269): tablet 493d11bbb3144ca1a466ada6fe5df2a0 (table test-table [id=99edacb2b4a8498280eb24a2b655cbda]) successfully deleted
W20260925 12:14:36.877048 14903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:36.877274 14903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:36.877298 14903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:36.878526 14903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:36.878566 14903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:36.878592 14903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:36.880559 14903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45227
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:36.880826 14903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:36.881115 14903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:14:36.886718 14903 server_base.cc:1034] running on GCE node
W20260925 12:14:36.886673 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.886662 14912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:36.886673 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:36.887144 14903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:36.887446 14903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:36.888572 14903 hybrid_clock.cc:648] HybridClock initialized: now 1790338476888552 us; error 49 us; skew 500 ppm
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:36.891738 14903 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:36.892181 14903 webserver.cc:460] Webserver started at http://127.0.98.2:45227/ using document root <none> and password file <none>
I20260925 12:14:36.892409 14903 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:36.892519 14903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:36.894425 14903 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:14:36.895224 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.896116 14903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:36.896217 14903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "86eb475ee97bd5b2406edbce01c64094"
server_key_iv: "a7decc6c693681f6a47461cb1cd02ce2"
server_key_version: "encryptionkey@0"
I20260925 12:14:36.896751 14903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:36.906472 14903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:36.908835 14903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:36.908995 14903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:36.909230 14903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:36.909536 14903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:36.909585 14903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.909628 14903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:36.909662 14903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:36.915822 14903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34427
I20260925 12:14:36.915896 15035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34427 every 8 connection(s)
I20260925 12:14:36.916247 14903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:14:36 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:36.924974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 14903
I20260925 12:14:36.925266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33333
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=36481
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:36.926013 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.917032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42561 (local address 127.0.98.62:35973)
0925 12:14:36.917095 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:36.917097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:36.917284 (+ 187us) server_negotiation.cc:407] Connection header received
0925 12:14:36.917777 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:36.917778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:36.917792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:36.917846 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:36.918828 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.919331 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.920286 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.920477 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.920605 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.922675 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:36.922677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:36.922677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:36.922684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:36.924151 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.924659 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.924660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.924660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.924719 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:36.925129 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:36.925130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:36.925131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:36.925295 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:36.925348 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:36.925764 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 12:14:36.925868 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 12:14:36.926151 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:36.917046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:42561)
0925 12:14:36.917177 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:36.917213 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:36.917686 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:36.917916 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:36.917920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:36.918541 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:36.918737 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.918742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.919410 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.919411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.920215 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:36.920218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:36.920617 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:36.920618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:36.920723 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:36.921192 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:36.921200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:36.922573 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:36.924271 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.924279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.924282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.924599 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.924850 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:36.924851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:36.924851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:36.925005 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:36.925402 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:36.925411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:36.925600 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 12:14:36.925724 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 12:14:36.925917 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260925 12:14:36.926641 15036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:36.926759 15036 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:36.926954 15036 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:36.927327 14710 ts_manager.cc:194] Registered new tserver with Master: 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427)
I20260925 12:14:36.927868 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42561
W20260925 12:14:37.011749 15039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:37.011978 15039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:37.012002 15039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:37.013202 15039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:37.013244 15039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:37.013270 15039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:37.014861 15039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36481
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:37.015120 15039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:37.015384 15039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:37.020545 15045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.020629 15044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.020567 15039 server_base.cc:1034] running on GCE node
W20260925 12:14:37.020545 15047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.020987 15039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:37.021235 15039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:37.022403 15039 hybrid_clock.cc:648] HybridClock initialized: now 1790338477022399 us; error 33 us; skew 500 ppm
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:37.024986 15039 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:37.025434 15039 webserver.cc:460] Webserver started at http://127.0.98.3:36481/ using document root <none> and password file <none>
I20260925 12:14:37.025667 15039 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:37.025725 15039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:37.027074 15039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:37.027709 15054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.027885 15039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.027973 15039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "720b5fc494def03cdf54c2f714ba301b"
server_key_iv: "7c32c68b8896bb230ae2da96fe298ef8"
server_key_version: "encryptionkey@0"
I20260925 12:14:37.028307 15039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:37.039354 15039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:37.041755 15039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:37.041924 15039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:37.042162 15039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:37.042482 15039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:37.042517 15039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.042557 15039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:37.042582 15039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.049013 15039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33333
I20260925 12:14:37.049077 15167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33333 every 8 connection(s)
I20260925 12:14:37.049422 15039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:14:37.050866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15039
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:37.059089 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.051275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.98.62:35973)
0925 12:14:37.051376 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.051378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.051508 (+ 130us) server_negotiation.cc:407] Connection header received
0925 12:14:37.051560 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.051561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.051573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.051643 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:37.052199 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.052992 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.054050 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.054362 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.054442 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.056121 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:37.056122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:37.056123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:37.056130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:37.057777 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:37.058162 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:37.058163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:37.058163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:37.058215 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:37.058453 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:37.058455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:37.058455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:37.058634 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:37.058818 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.058932 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.059010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
I20260925 12:14:37.060992 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.050148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48879 (local address 127.0.98.62:35973)
0925 12:14:37.050214 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.050215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.050523 (+ 308us) server_negotiation.cc:407] Connection header received
0925 12:14:37.051429 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.051430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.051445 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.051514 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:37.052758 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.053560 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.054888 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.055171 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.055235 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.057655 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:37.057657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:37.057657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:37.057665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:37.059448 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:37.059940 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:37.059940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:37.059941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:37.060023 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:37.060331 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:37.060332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:37.060332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:37.060454 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:37.060639 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.060859 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.060928 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:14:37.061188 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.050145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:48879)
0925 12:14:37.050325 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.050437 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.051163 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.051607 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:37.051612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:37.052303 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:37.052650 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.052656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.053647 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.053648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.054808 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.054811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.055260 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.055261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.055420 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.055954 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:37.055965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:37.057559 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:37.059528 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:37.059554 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:37.059557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:37.059848 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:37.060124 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:37.060125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:37.060126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:37.060240 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:37.060589 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:37.060594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:37.060732 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 12:14:37.060830 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 12:14:37.060978 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 12:14:37.061745 15168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35973
I20260925 12:14:37.061833 15168 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:37.062014 15168 heartbeater.cc:511] Master 127.0.98.62:35973 requested a full tablet report, sending...
I20260925 12:14:37.062386 14710 ts_manager.cc:194] Registered new tserver with Master: 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
I20260925 12:14:37.062722 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48879
I20260925 12:14:37.071012 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:14:37.074160 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.071458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.0.98.62:35973)
0925 12:14:37.071502 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.071519 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.071648 (+ 129us) server_negotiation.cc:407] Connection header received
0925 12:14:37.071755 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.071756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.071794 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.071871 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:14:37.072004 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.072564 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.073360 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.073559 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.073645 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.073812 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:14:37.073931 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:14:37.074030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.074048 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.074067 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:14:37.074470 14710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:14:37.076151 14710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:14:37.078843 14832 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7df731c874f7f82df82b7ab93272c (DEFAULT_TABLE table=test-table [id=830e4bfde8fd4d4b9eb7340c54ba1d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:37.079063 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7df731c874f7f82df82b7ab93272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:37.080694 14790 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap starting.
I20260925 12:14:37.081022 14790 tablet_bootstrap.cc:654] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:37.081585 14790 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: No bootstrap required, opened a new log
I20260925 12:14:37.081636 14790 ts_tablet_manager.cc:1397] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:37.081993 14790 raft_consensus.cc:348] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.082083 14790 raft_consensus.cc:374] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:37.082135 14790 raft_consensus.cc:729] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b2d474d05143a3a07d1b5b8faf085b, State: Initialized, Role: FOLLOWER
I20260925 12:14:37.082185 14790 consensus_queue.cc:260] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.082286 14790 ts_tablet_manager.cc:1428] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:37.083443 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.078270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33333 (local address 127.0.0.1:33588)
0925 12:14:37.078426 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.078437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.078485 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.078877 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:37.078879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:37.078881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:37.078982 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.078984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.080335 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.080336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.081912 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.081916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.083234 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.083235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.083288 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.083303 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 12:14:37.083343 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:14:37.083378 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260925 12:14:37.083492 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.078401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.98.3:33333)
0925 12:14:37.078643 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.078649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.078661 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:14:37.078708 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.078711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.078729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.078826 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:37.079095 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.080214 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.082025 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.083154 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.083235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.083293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.083366 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.083406 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260925 12:14:37.084242 15102 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7df731c874f7f82df82b7ab93272c (DEFAULT_TABLE table=test-table [id=830e4bfde8fd4d4b9eb7340c54ba1d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:37.084635 15102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7df731c874f7f82df82b7ab93272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:37.086388 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.078171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34427 (local address 127.0.0.1:53778)
0925 12:14:37.078228 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.078241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.078292 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.082641 (+ 4349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:37.082642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:37.082644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:37.082687 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.082688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.083974 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.083975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.085059 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.085063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.086116 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.086117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.086194 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.086200 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:14:37.086272 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 12:14:37.086322 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:14:37.086433 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.079291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.98.2:34427)
0925 12:14:37.082423 (+ 3132us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.082429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.082442 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:14:37.082484 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.082486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.082513 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.082754 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:37.082827 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.083863 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.085153 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.086022 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.086106 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.086170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.086288 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.086337 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2128,"thread_start_us":74,"threads_started":1}
I20260925 12:14:37.087745 15178 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Bootstrap starting.
I20260925 12:14:37.088217 14970 tablet_service.cc:1467] Processing CreateTablet for tablet e3f7df731c874f7f82df82b7ab93272c (DEFAULT_TABLE table=test-table [id=830e4bfde8fd4d4b9eb7340c54ba1d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:37.088387 15178 tablet_bootstrap.cc:654] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:37.088553 14970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f7df731c874f7f82df82b7ab93272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:37.088999 15178 log.cc:824] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:37.089700 15178 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: No bootstrap required, opened a new log
I20260925 12:14:37.089784 15178 ts_tablet_manager.cc:1397] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:37.091243 15180 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Bootstrap starting.
I20260925 12:14:37.091144 15178 raft_consensus.cc:348] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.091323 15178 raft_consensus.cc:374] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:37.091367 15178 raft_consensus.cc:729] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365281ddf2d14ea9b855fc7dda1e9c5e, State: Initialized, Role: FOLLOWER
I20260925 12:14:37.091519 15178 consensus_queue.cc:260] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.091861 15178 ts_tablet_manager.cc:1428] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:37.091898 15168 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
I20260925 12:14:37.091964 15180 tablet_bootstrap.cc:654] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:37.092594 15180 log.cc:824] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:37.093338 15180 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: No bootstrap required, opened a new log
I20260925 12:14:37.093393 15180 ts_tablet_manager.cc:1397] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:14:37.094684 15180 raft_consensus.cc:348] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.094853 15180 raft_consensus.cc:374] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:37.094894 15180 raft_consensus.cc:729] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998f8a1a6f264f2388c0711ecb8bb0fd, State: Initialized, Role: FOLLOWER
I20260925 12:14:37.095017 15180 consensus_queue.cc:260] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.095316 15180 ts_tablet_manager.cc:1428] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:14:37.095346 15036 heartbeater.cc:503] Master 127.0.98.62:35973 was elected leader, sending a full tablet report...
W20260925 12:14:37.166913 15037 tablet.cc:2369] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:14:37.187536 15181 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:37.187674 15181 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.187989 15181 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269)
I20260925 12:14:37.190846 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.188340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40319 (local address 127.0.98.2:34427)
0925 12:14:37.188389 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.188391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.188404 (+ 13us) server_negotiation.cc:407] Connection header received
0925 12:14:37.188437 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.188437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.188449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.188562 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:37.188729 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.189202 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190122 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.190500 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190582 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.190593 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.190745 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.190779 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:14:37.190848 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.188213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34427 (local address 127.0.98.3:40319)
0925 12:14:37.188298 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.188315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.188381 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.188572 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:37.188573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:37.188574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:37.188649 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.188650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.189285 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.189286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.190031 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.190033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.190580 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.190664 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.190667 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:14:37.190727 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:14:37.190772 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 12:14:37.191164 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.98.3:57671)
0925 12:14:37.188522 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.188531 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.188591 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.188782 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:37.188783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:37.188784 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:37.188887 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.188889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.189607 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.189608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.190360 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:37.190362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.190919 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:37.190985 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.190989 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:37.191045 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:14:37.191084 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260925 12:14:37.191159 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.188445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57671 (local address 127.0.98.1:37269)
0925 12:14:37.188512 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.188514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.188651 (+ 137us) server_negotiation.cc:407] Connection header received
0925 12:14:37.188685 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.188686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.188708 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.188779 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:37.188963 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.189520 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190452 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.190840 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.190919 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.191005 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.191061 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.191100 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:14:37.191192 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7df731c874f7f82df82b7ab93272c" candidate_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" is_pre_election: true
I20260925 12:14:37.191349 14990 raft_consensus.cc:2454] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365281ddf2d14ea9b855fc7dda1e9c5e in term 0.
I20260925 12:14:37.191548 14852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7df731c874f7f82df82b7ab93272c" candidate_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" is_pre_election: true
I20260925 12:14:37.191601 15058 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd; no voters:
I20260925 12:14:37.191691 14852 raft_consensus.cc:2454] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365281ddf2d14ea9b855fc7dda1e9c5e in term 0.
I20260925 12:14:37.191744 15181 raft_consensus.cc:2781] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:14:37.191816 15181 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:14:37.191851 15181 raft_consensus.cc:3037] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:37.192582 15181 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.192744 15181 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [CANDIDATE]: Term 1 election: Requested vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269)
I20260925 12:14:37.192900 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7df731c874f7f82df82b7ab93272c" candidate_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
I20260925 12:14:37.192924 14852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f7df731c874f7f82df82b7ab93272c" candidate_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
I20260925 12:14:37.192992 14852 raft_consensus.cc:3037] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:37.192988 14990 raft_consensus.cc:3037] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:37.193894 14990 raft_consensus.cc:2454] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365281ddf2d14ea9b855fc7dda1e9c5e in term 1.
I20260925 12:14:37.193894 14852 raft_consensus.cc:2454] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365281ddf2d14ea9b855fc7dda1e9c5e in term 1.
I20260925 12:14:37.194070 15058 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd; no voters:
I20260925 12:14:37.194166 15181 raft_consensus.cc:2781] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:37.194315 15181 raft_consensus.cc:686] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 LEADER]: Becoming Leader. State: Replica: 365281ddf2d14ea9b855fc7dda1e9c5e, State: Running, Role: LEADER
I20260925 12:14:37.194422 15181 consensus_queue.cc:237] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.195207 14710 catalog_manager.cc:5495] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } health_report { overall_health: UNKNOWN } } }
I20260925 12:14:37.203421 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.201124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.98.3:33333)
0925 12:14:37.201174 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:37.201176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:37.201214 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:14:37.201280 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:37.201281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:37.201298 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:37.201357 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:14:37.201457 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.201942 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.202648 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:37.202862 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:37.202927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:37.203043 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:14:37.203143 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:14:37.203203 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:37.203219 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 12:14:37.203234 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:14:37.205260 14852 raft_consensus.cc:1261] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Refusing update from remote peer 365281ddf2d14ea9b855fc7dda1e9c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:37.205292 14990 raft_consensus.cc:1261] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Refusing update from remote peer 365281ddf2d14ea9b855fc7dda1e9c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:37.205654 15185 consensus_queue.cc:1035] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:37.205793 15181 consensus_queue.cc:1035] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:37.210107 14710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:37.210215 14710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:37.210925 14710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Sending DeleteTablet for 3 replicas of tablet e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:37.211337 14970 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f7df731c874f7f82df82b7ab93272c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53778
I20260925 12:14:37.211392 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f7df731c874f7f82df82b7ab93272c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47796
I20260925 12:14:37.211400 15102 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f7df731c874f7f82df82b7ab93272c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33588
I20260925 12:14:37.211563 14907 tablet_replica.cc:330] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: stopping tablet replica
I20260925 12:14:37.211575 15197 tablet_replica.cc:330] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: stopping tablet replica
I20260925 12:14:37.211620 14907 raft_consensus.cc:2229] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:37.211671 15197 raft_consensus.cc:2229] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:37.211684 14907 raft_consensus.cc:2258] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:37.211782 15197 raft_consensus.cc:2258] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:37.211962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14748
I20260925 12:14:37.212875 15197 ts_tablet_manager.cc:1905] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:37.215091 15197 ts_tablet_manager.cc:1918] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:37.215175 15197 log.cc:1196] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:37.215512 15197 ts_tablet_manager.cc:1939] T e3f7df731c874f7f82df82b7ab93272c P 998f8a1a6f264f2388c0711ecb8bb0fd: Deleting consensus metadata
I20260925 12:14:37.215993 14697 catalog_manager.cc:4841] TS 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): tablet e3f7df731c874f7f82df82b7ab93272c (table test-table [id=830e4bfde8fd4d4b9eb7340c54ba1d9f]) successfully deleted
W20260925 12:14:37.218537 14697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37269 (error 108)
I20260925 12:14:37.218915 15198 tablet_replica.cc:330] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: stopping tablet replica
I20260925 12:14:37.218909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14903
I20260925 12:14:37.220335 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.220150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37269 (local address 127.0.0.1:47810)
0925 12:14:37.220227 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.220283 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 12:14:37.220383 15198 raft_consensus.cc:2229] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:14:37.220535 15198 raft_consensus.cc:2258] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:37.220988 15198 ts_tablet_manager.cc:1905] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 12:14:37.220469 14697 catalog_manager.cc:4584] TS d9b2d474d05143a3a07d1b5b8faf085b (127.0.98.1:37269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3f7df731c874f7f82df82b7ab93272c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37269: connect: Connection refused (error 111)
I20260925 12:14:37.222762 15198 ts_tablet_manager.cc:1918] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:37.222855 15198 log.cc:1196] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:37.223119 15198 ts_tablet_manager.cc:1939] T e3f7df731c874f7f82df82b7ab93272c P 365281ddf2d14ea9b855fc7dda1e9c5e: Deleting consensus metadata
I20260925 12:14:37.223554 14697 catalog_manager.cc:4841] TS 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): tablet e3f7df731c874f7f82df82b7ab93272c (table test-table [id=830e4bfde8fd4d4b9eb7340c54ba1d9f]) successfully deleted
I20260925 12:14:37.224570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15039
I20260925 12:14:37.229563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 14677
I20260925 12:14:37.234961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--webserver_interface=127.0.98.62
--webserver_port=33047
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:37.316876 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:37.317101 15200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:37.317123 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:37.318269 15200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:37.318306 15200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:37.318321 15200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:37.318334 15200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:37.318346 15200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:37.319885 15200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:37.320091 15200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:37.320279 15200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:37.325310 15205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.325349 15200 server_base.cc:1034] running on GCE node
W20260925 12:14:37.325294 15206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.325290 15208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.325750 15200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:37.326088 15200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:37.327240 15200 hybrid_clock.cc:648] HybridClock initialized: now 1790338477327230 us; error 31 us; skew 500 ppm
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:37.330008 15200 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:37.330543 15200 webserver.cc:460] Webserver started at http://127.0.98.62:33047/ using document root <none> and password file <none>
I20260925 12:14:37.330797 15200 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:37.330852 15200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:37.332278 15200 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:37.333099 15215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.333308 15200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:37.333421 15200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccc95361f1af4e758e08486b6750ab40"
format_stamp: "Formatted at 2026-09-25 12:14:35 on dist-test-slave-2hqq"
server_key: "547cecf2859ac4a7b4becbd1d1cda543"
server_key_iv: "26749bbe258e2b5a2238631ba3c02122"
server_key_version: "encryptionkey@0"
I20260925 12:14:37.333796 15200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:37.352013 15200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:37.354539 15200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:37.354722 15200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:37.359155 15200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35973
I20260925 12:14:37.359187 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35973 every 8 connection(s)
I20260925 12:14:37.359601 15200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:37.360128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15200
I20260925 12:14:37.360427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37269
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=45945
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:14:37.361439 15268 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:14:37.362486 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap starting.
I20260925 12:14:37.373600 15268 log.cc:824] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:37.376638 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:14:37.376999 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40: Bootstrap complete.
I20260925 12:14:37.380259 15268 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:37.380637 15268 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Initialized, Role: FOLLOWER
I20260925 12:14:37.380817 15268 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:37.380880 15268 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:37.380925 15268 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:37.380975 15268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 2 FOLLOWER]: Advancing to term 3
I20260925 12:14:37.382509 15268 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:37.382629 15268 leader_election.cc:302] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc95361f1af4e758e08486b6750ab40; no voters:
I20260925 12:14:37.382798 15268 leader_election.cc:288] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 12:14:37.382870 15272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 12:14:37.383059 15268 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:37.383069 15272 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [term 3 LEADER]: Becoming Leader. State: Replica: ccc95361f1af4e758e08486b6750ab40, State: Running, Role: LEADER
I20260925 12:14:37.383322 15272 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } }
I20260925 12:14:37.383669 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:37.383754 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:37.383913 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc95361f1af4e758e08486b6750ab40. Latest consensus state: current_term: 3 leader_uuid: "ccc95361f1af4e758e08486b6750ab40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc95361f1af4e758e08486b6750ab40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35973 } } }
I20260925 12:14:37.384022 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc95361f1af4e758e08486b6750ab40 [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:37.384269 15281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:37.385174 15281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:14:37.385279 15281 catalog_manager.cc:1259] Loaded cluster ID: a499f14429c44fa8a1a43a7a556e110a
I20260925 12:14:37.385311 15281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 12:14:37.385542 15281 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0a0bfc79ad google::LogMessage::Fail() at ??:0
@ 0x7f0a0bfca88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0a0bfc74de google::LogMessage::Flush() at ??:0
@ 0x7f0a0bfc8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0a13ee80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0a13ef18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0a0de950d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0a0de8a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0a0ecee6db start_thread at ??:0
@ 0x7f0a0aac561f clone at ??:0
W20260925 12:14:37.465059 15270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:37.465278 15270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:37.465305 15270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:37.466531 15270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:37.466576 15270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:37.466603 15270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:37.468201 15270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45945
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:37.468474 15270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:37.468771 15270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:37.474291 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.474287 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.474542 15270 server_base.cc:1034] running on GCE node
W20260925 12:14:37.474673 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.474877 15270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:37.475170 15270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:37.476336 15270 hybrid_clock.cc:648] HybridClock initialized: now 1790338477476318 us; error 37 us; skew 500 ppm
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:37.478976 15270 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:37.479468 15270 webserver.cc:460] Webserver started at http://127.0.98.1:45945/ using document root <none> and password file <none>
I20260925 12:14:37.479715 15270 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:37.479761 15270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:37.481294 15270 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 12:14:37.482177 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.482358 15270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:37.482453 15270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "d9b2d474d05143a3a07d1b5b8faf085b"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "dd85d03a5e74959bd473186a51245bac"
server_key_iv: "e5ce69b64308ed87885b4bde4a88ebac"
server_key_version: "encryptionkey@0"
I20260925 12:14:37.482831 15270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:37.524564 15270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:37.527319 15270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:37.527493 15270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:37.527776 15270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:37.528317 15321 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:14:37.529932 15270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:14:37.529987 15270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:14:37.530014 15270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:14:37.530755 15270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:14:37.530798 15270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:14:37.530905 15321 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap starting.
I20260925 12:14:37.538141 15270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37269
I20260925 12:14:37.538612 15270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:14:37.540624 15428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37269 every 8 connection(s)
I20260925 12:14:37.541874 15321 log.cc:824] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:37.544094 15321 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:14:37.544386 15321 tablet_bootstrap.cc:492] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Bootstrap complete.
I20260925 12:14:37.545383 15321 ts_tablet_manager.cc:1397] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260925 12:14:37.546453 15321 raft_consensus.cc:348] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.546732 15321 raft_consensus.cc:729] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b2d474d05143a3a07d1b5b8faf085b, State: Initialized, Role: FOLLOWER
I20260925 12:14:37.546897 15321 consensus_queue.cc:260] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:37.547232 15321 ts_tablet_manager.cc:1428] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 12:14:37.548659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15270
I20260925 12:14:37.548938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34427
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=45227
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:37.597729 15431 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:14:37.539295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:43039)
0925 12:14:37.539488 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.539532 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:37.540264 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:37.597515 (+ 57251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
W20260925 12:14:37.598049 15429 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35973 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35973: Client connection negotiation failed: client connection to 127.0.98.62:35973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:14:37.598434 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.598315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:41465)
0925 12:14:37.598369 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.598412 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:14:37.598829 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.598735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:35381)
0925 12:14:37.598775 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.598810 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 12:14:37.598975 15429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:14:37.635021 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:37.635231 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:37.635257 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:37.636557 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:37.636601 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:37.636629 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:37.638211 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45227
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:37.638470 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:37.638706 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:37.644029 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.644021 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.644099 15434 server_base.cc:1034] running on GCE node
W20260925 12:14:37.644035 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.644433 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:37.644714 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:37.645879 15434 hybrid_clock.cc:648] HybridClock initialized: now 1790338477645865 us; error 28 us; skew 500 ppm
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:37.648411 15434 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:37.648897 15434 webserver.cc:460] Webserver started at http://127.0.98.2:45227/ using document root <none> and password file <none>
I20260925 12:14:37.649101 15434 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:37.649142 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:37.650434 15434 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:37.651037 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.651221 15434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:37.651301 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "998f8a1a6f264f2388c0711ecb8bb0fd"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "86eb475ee97bd5b2406edbce01c64094"
server_key_iv: "a7decc6c693681f6a47461cb1cd02ce2"
server_key_version: "encryptionkey@0"
I20260925 12:14:37.651648 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:37.671897 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:37.674007 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:37.674129 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:37.674377 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:37.674718 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:37.674768 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.674804 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:37.674818 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.680856 15434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34427
I20260925 12:14:37.680974 15562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34427 every 8 connection(s)
I20260925 12:14:37.681314 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:14:37.682518 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.682047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:56033)
0925 12:14:37.682241 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.682374 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
W20260925 12:14:37.682958 15563 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35973 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35973: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
I20260925 12:14:37.683251 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.683162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:50605)
0925 12:14:37.683201 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.683228 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 12:14:37.683537 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.683466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:58053)
0925 12:14:37.683502 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.683523 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 12:14:37.683650 15563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:14:37.684427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15434
I20260925 12:14:37.684689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33333
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35973
--webserver_port=36481
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:37.766790 15566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:37.767015 15566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:37.767041 15566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:37.768183 15566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:37.768224 15566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:37.768249 15566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:37.769744 15566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36481
--tserver_master_addrs=127.0.98.62:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:37.769939 15566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:37.770133 15566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:37.775588 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.775461 15572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:37.775496 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:37.775868 15566 server_base.cc:1034] running on GCE node
I20260925 12:14:37.776170 15566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:37.776432 15566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:37.777635 15566 hybrid_clock.cc:648] HybridClock initialized: now 1790338477777622 us; error 30 us; skew 500 ppm
Sep 25 12:14:37 dist-test-slave-2hqq krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:37.780445 15566 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:37.780961 15566 webserver.cc:460] Webserver started at http://127.0.98.3:36481/ using document root <none> and password file <none>
I20260925 12:14:37.781222 15566 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:37.781275 15566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:37.782617 15566 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:37.783172 15581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.783334 15566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:14:37.783416 15566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "365281ddf2d14ea9b855fc7dda1e9c5e"
format_stamp: "Formatted at 2026-09-25 12:14:36 on dist-test-slave-2hqq"
server_key: "720b5fc494def03cdf54c2f714ba301b"
server_key_iv: "7c32c68b8896bb230ae2da96fe298ef8"
server_key_version: "encryptionkey@0"
I20260925 12:14:37.783800 15566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:37.790817 15566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:37.792939 15566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:37.793052 15566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:37.793231 15566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:37.793542 15566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:37.793596 15566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.793637 15566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:37.793661 15566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:37.799192 15566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33333
I20260925 12:14:37.799265 15694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33333 every 8 connection(s)
I20260925 12:14:37.799616 15566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:14:37.800793 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.800320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:53753)
0925 12:14:37.800578 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.800650 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
W20260925 12:14:37.801183 15695 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35973 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35973: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
I20260925 12:14:37.801476 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.801383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:52875)
0925 12:14:37.801432 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.801459 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:14:37.801697 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:37.801651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:60243)
0925 12:14:37.801669 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:37.801686 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260925 12:14:37.801810 15695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:14:37.809664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15566
I20260925 12:14:38.599752 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:38.599448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:48073)
0925 12:14:38.599632 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:38.599686 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260925 12:14:38.684216 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:38.684004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:38751)
0925 12:14:38.684120 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:38.684161 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 12:14:38.802304 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:38.802105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:55579)
0925 12:14:38.802219 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:38.802250 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260925 12:14:39.336148 15702 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:39.336279 15702 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:39.336676 15702 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
Sep 25 12:14:39 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 12:14:39 dist-test-slave-2hqq krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 12:14:39.347255 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.337333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45121 (local address 127.0.98.3:33333)
0925 12:14:39.337466 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:39.337473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:39.337493 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:14:39.337987 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:39.337990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:39.338051 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:39.338196 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:39.338910 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.339745 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.340785 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.341180 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.341291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:39.344023 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:39.344027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:39.344027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:39.344038 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:39.345735 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346096 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:39.346098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:39.346098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:39.346198 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346416 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:39.346417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:39.346417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:39.346638 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:39.346730 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:39.347024 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 12:14:39.347148 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260925 12:14:39.347332 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.336875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34427 (local address 127.0.98.1:37177)
0925 12:14:39.337002 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:39.337014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:39.337179 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:39.338254 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:39.338257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:39.338579 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:39.338787 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:39.338792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.339875 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.339876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:39.340700 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:39.340703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.341292 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.341293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:39.341372 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:39.341990 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:39.341995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:39.343536 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:39.345642 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:39.345649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:39.345651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:39.345961 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:39.346212 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:39.346213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:39.346336 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:39.346707 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:39.346714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:39.346844 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 12:14:39.346951 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 12:14:39.347145 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":72,"thread_start_us":66,"threads_started":1}
I20260925 12:14:39.347275 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.336985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37177 (local address 127.0.98.2:34427)
0925 12:14:39.337318 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:39.337330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:39.337360 (+ 30us) server_negotiation.cc:407] Connection header received
0925 12:14:39.337987 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:39.337990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:39.338051 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:39.338196 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:39.338899 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.339754 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.340825 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.341198 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.341323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:39.343652 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:39.343655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:39.343656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:39.343666 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:39.345559 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346033 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:39.346037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:39.346037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:39.346146 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346398 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:39.346399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:39.346400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:39.346639 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:39.346730 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:39.347004 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 12:14:39.347134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":66,"threads_started":1}
I20260925 12:14:39.347926 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.337136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33333 (local address 127.0.98.1:45121)
0925 12:14:39.337249 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:39.337260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:39.337322 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:39.338332 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:39.338333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:39.338582 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:39.338787 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:39.338792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.339843 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.339844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:39.340702 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:39.340705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:39.341263 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:39.341264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:39.341372 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:39.341940 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:39.341948 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:39.343918 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:39.345817 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:39.345818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:39.345818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:39.346022 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:39.346269 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:39.346270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:39.346270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:39.346339 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:39.346731 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:39.346732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:39.346844 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 12:14:39.346963 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 12:14:39.347225 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":135,"thread_start_us":52,"threads_started":1}
W20260925 12:14:39.348251 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:39.348551 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:39.348603 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:39.348740 15702 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:39.600603 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.600414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:53177)
0925 12:14:39.600513 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:39.600561 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 12:14:39.684908 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.684702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:34777)
0925 12:14:39.684817 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:39.684864 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260925 12:14:39.802948 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:39.802728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:46951)
0925 12:14:39.802861 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:39.802908 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 12:14:40.601476 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:40.601231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:40487)
0925 12:14:40.601392 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:40.601428 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 12:14:40.685626 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:40.685401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:43483)
0925 12:14:40.685512 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:40.685557 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260925 12:14:40.803651 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:40.803425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:45721)
0925 12:14:40.803570 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:40.803608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260925 12:14:41.077386 15712 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:41.077499 15712 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:41.077679 15712 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:41.078209 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:41.078310 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:41.078353 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:41.078450 15712 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:41.602147 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:41.601917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:34449)
0925 12:14:41.602061 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:41.602101 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260925 12:14:41.686270 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:41.686074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:57203)
0925 12:14:41.686190 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:41.686230 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260925 12:14:41.804365 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:41.804162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:38291)
0925 12:14:41.804302 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:41.804330 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260925 12:14:42.603012 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:42.602728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:43069)
0925 12:14:42.602919 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:42.602963 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":135,"threads_started":1}
I20260925 12:14:42.687043 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:42.686835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:48739)
0925 12:14:42.686961 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:42.687001 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260925 12:14:42.805033 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:42.804833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:43491)
0925 12:14:42.804960 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:42.804996 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260925 12:14:43.603781 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:43.603555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:40351)
0925 12:14:43.603691 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:43.603732 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260925 12:14:43.687722 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:43.687481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:32771)
0925 12:14:43.687638 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:43.687678 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260925 12:14:43.805691 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:43.805501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:50321)
0925 12:14:43.805611 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:43.805648 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260925 12:14:44.287592 15722 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:44.287750 15722 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:44.287945 15722 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:44.288363 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:44.288486 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:44.288539 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:44.288631 15722 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:44.604754 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:44.604358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:36645)
0925 12:14:44.604652 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:44.604707 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260925 12:14:44.688585 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:44.688282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:42029)
0925 12:14:44.688419 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:44.688476 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260925 12:14:44.806459 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:44.806245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:50027)
0925 12:14:44.806374 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:44.806409 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 12:14:45.605629 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:45.605337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:41895)
0925 12:14:45.605527 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:45.605575 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260925 12:14:45.689252 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:45.689037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:41247)
0925 12:14:45.689166 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:45.689206 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260925 12:14:45.807194 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:45.806916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:43907)
0925 12:14:45.807098 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:45.807141 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260925 12:14:46.606653 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:46.606336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:59703)
0925 12:14:46.606557 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:46.606603 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":132,"threads_started":1}
I20260925 12:14:46.690022 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:46.689777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:45321)
0925 12:14:46.689931 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:46.689976 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 12:14:46.807977 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:46.807712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:39869)
0925 12:14:46.807883 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:46.807931 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260925 12:14:47.607877 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:47.607471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:47571)
0925 12:14:47.607754 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:47.607788 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":81,"threads_started":1}
I20260925 12:14:47.690814 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:47.690562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:34239)
0925 12:14:47.690714 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:47.690762 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260925 12:14:47.808861 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:47.808629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:39451)
0925 12:14:47.808770 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:47.808815 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260925 12:14:48.608821 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:48.608552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:37509)
0925 12:14:48.608732 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:48.608771 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260925 12:14:48.680581 15736 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:48.680708 15736 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:48.680935 15736 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:48.681443 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:48.681548 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:48.681582 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:48.681694 15736 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:48.691551 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:48.691317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:33541)
0925 12:14:48.691470 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:48.691508 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260925 12:14:48.809788 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:48.809506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:41411)
0925 12:14:48.809691 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:48.809739 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260925 12:14:49.609649 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:49.609385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:60289)
0925 12:14:49.609542 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:49.609588 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260925 12:14:49.692584 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:49.692289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:35599)
0925 12:14:49.692430 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:49.692479 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 12:14:49.810673 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:49.810431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:36965)
0925 12:14:49.810571 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:49.810616 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260925 12:14:50.610440 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:50.610223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:35835)
0925 12:14:50.610350 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:50.610393 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 12:14:50.693338 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:50.693087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:33749)
0925 12:14:50.693248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:50.693293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260925 12:14:50.811494 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:50.811234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:40769)
0925 12:14:50.811396 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:50.811450 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
I20260925 12:14:50.899395 15745 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:50.899529 15745 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:50.899717 15745 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:50.900120 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:50.900195 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:50.900210 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:50.900287 15745 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:51.611269 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:51.610986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:36735)
0925 12:14:51.611153 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:51.611194 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260925 12:14:51.694125 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:51.693852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:48943)
0925 12:14:51.694035 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:51.694075 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260925 12:14:51.812336 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:51.812074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:35479)
0925 12:14:51.812253 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:51.812293 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260925 12:14:52.612229 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:52.611968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.1:43347)
0925 12:14:52.612129 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:52.612182 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260925 12:14:52.686784 15750 raft_consensus.cc:482] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:52.687009 15750 raft_consensus.cc:504] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998f8a1a6f264f2388c0711ecb8bb0fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34427 } } peers { permanent_uuid: "365281ddf2d14ea9b855fc7dda1e9c5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33333 } } peers { permanent_uuid: "d9b2d474d05143a3a07d1b5b8faf085b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37269 } }
I20260925 12:14:52.687215 15750 leader_election.cc:288] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427), 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333)
W20260925 12:14:52.687659 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 365281ddf2d14ea9b855fc7dda1e9c5e (127.0.98.3:33333): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
W20260925 12:14:52.687737 15318 leader_election.cc:341] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 998f8a1a6f264f2388c0711ecb8bb0fd (127.0.98.2:34427): Not found: Tablet not found: e3f7df731c874f7f82df82b7ab93272c
I20260925 12:14:52.687789 15318 leader_election.cc:302] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9b2d474d05143a3a07d1b5b8faf085b; no voters: 365281ddf2d14ea9b855fc7dda1e9c5e, 998f8a1a6f264f2388c0711ecb8bb0fd
I20260925 12:14:52.687849 15750 raft_consensus.cc:2726] T e3f7df731c874f7f82df82b7ab93272c P d9b2d474d05143a3a07d1b5b8faf085b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:14:52.694965 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:52.694740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.2:43561)
0925 12:14:52.694884 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:52.694915 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260925 12:14:52.813177 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:52.812939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35973 (local address 127.0.98.3:47961)
0925 12:14:52.813083 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:52.813123 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260925 12:14:52.814368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15270
I20260925 12:14:52.819274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15434
I20260925 12:14:52.823645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15566
2026-09-25T12:14:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:14:52 dist-test-slave-2hqq krb5kdc[15755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:14:52 dist-test-slave-2hqq krb5kdc[15755](info): set up 2 sockets
Sep 25 12:14:52 dist-test-slave-2hqq krb5kdc[15755](info): commencing operation
krb5kdc: starting...
W20260925 12:14:53.884800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:14:53 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:14:54.011586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:54.093245 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:54.093502 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:54.093541 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:54.094789 15768 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:14:54.094873 15768 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:14:54.094894 15768 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:14:54.094913 15768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:14:54.094939 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:54.094971 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:54.094995 15768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:14:54.095024 15768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:14:54.096576 15768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:54.096891 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:54.097174 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:54.102603 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.102751 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.102603 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:54.102866 15768 server_base.cc:1034] running on GCE node
I20260925 12:14:54.103103 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:54.103446 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:54.104640 15768 hybrid_clock.cc:648] HybridClock initialized: now 1790338494104597 us; error 70 us; skew 500 ppm
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:54.107390 15768 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:14:54.107867 15768 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260925 12:14:54.108122 15768 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:54.108206 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:54.108366 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:54.109627 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "ccf5fd333c0b49f1b34974882c96204f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "dad1f6a77dc808721c3f2733091bf3f6"
server_key_iv: "3b604745c8ae4ef7a5bf9b8e608e9a35"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.110028 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccf5fd333c0b49f1b34974882c96204f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "dad1f6a77dc808721c3f2733091bf3f6"
server_key_iv: "3b604745c8ae4ef7a5bf9b8e608e9a35"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.111511 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:14:54.112347 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.112586 15768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:14:54.112707 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccf5fd333c0b49f1b34974882c96204f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "dad1f6a77dc808721c3f2733091bf3f6"
server_key_iv: "3b604745c8ae4ef7a5bf9b8e608e9a35"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.112802 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:54.125008 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:54.127377 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:54.127553 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:54.131639 15768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43811
I20260925 12:14:54.131670 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43811 every 8 connection(s)
I20260925 12:14:54.132042 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:14:54.133035 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:54.135304 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap starting.
I20260925 12:14:54.135954 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:54.136255 15836 log.cc:824] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:54.136590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15768
I20260925 12:14:54.136693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:14:54.136880 392 external_mini_cluster.cc:1428] Setting key f0fbdc8d57e2225836150d192331d9dc
I20260925 12:14:54.137028 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: No bootstrap required, opened a new log
I20260925 12:14:54.139258 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:14:54.139456 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:54.139498 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Initialized, Role: FOLLOWER
I20260925 12:14:54.139640 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:14:54.139747 15836 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:14:54.139784 15836 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:14:54.139830 15836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:54.140578 15836 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:14:54.140697 15836 leader_election.cc:302] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf5fd333c0b49f1b34974882c96204f; no voters:
I20260925 12:14:54.140841 15836 leader_election.cc:288] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:14:54.140908 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:54.141050 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 LEADER]: Becoming Leader. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Running, Role: LEADER
I20260925 12:14:54.141060 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:14:54.141871 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:14:54.142511 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf5fd333c0b49f1b34974882c96204f. Latest consensus state: current_term: 1 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:14:54.142642 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:14:54.143057 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
I20260925 12:14:54.143072 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
W20260925 12:14:54.143349 15855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:14:54.143411 15855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:14:54.143884 15859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:14:54.144076 15859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:54.145936 15859 catalog_manager.cc:1347] Generated new cluster ID: 30115476a8f74610bf251db35d828cb7
I20260925 12:14:54.145987 15859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:14:54.150178 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.137420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.98.62:43811)
0925 12:14:54.137610 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.137620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.137655 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:14:54.138496 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.138505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.138570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.138724 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:54.139355 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.141138 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.142346 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.142936 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.143047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.146014 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:54.146019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:54.146019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:54.146030 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:54.148014 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.148403 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.148406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.148406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.148565 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.148830 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.148831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.148831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.149137 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:54.149320 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.149824 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.149964 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 12:14:54.162429 15859 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:14:54.163014 15859 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:14:54.166307 15859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Generated new TSK 0
I20260925 12:14:54.166502 15859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:14:54.203894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:54.303685 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:54.303928 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:54.303968 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:54.305198 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:54.305263 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:54.305308 15863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:14:54.306759 15863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:54.307025 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:54.307258 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:54.312463 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.312577 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.312546 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:54.312927 15863 server_base.cc:1034] running on GCE node
I20260925 12:14:54.313138 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:54.313373 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:54.314532 15863 hybrid_clock.cc:648] HybridClock initialized: now 1790338494314533 us; error 44 us; skew 500 ppm
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:54.317291 15863 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:14:54.317764 15863 webserver.cc:460] Webserver started at http://127.0.98.1:38397/ using document root <none> and password file <none>
I20260925 12:14:54.317986 15863 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:54.318060 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:54.318209 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:54.319309 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "be01383017b64a0e9ffbd37159671917"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "34fa8e254ac2f1b5694e2b1e3bef8dc0"
server_key_iv: "86c47ded8d8891810b8c8144efbc5257"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.319693 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be01383017b64a0e9ffbd37159671917"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "34fa8e254ac2f1b5694e2b1e3bef8dc0"
server_key_iv: "86c47ded8d8891810b8c8144efbc5257"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.321098 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:14:54.321873 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.322043 15863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:54.322140 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "be01383017b64a0e9ffbd37159671917"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "34fa8e254ac2f1b5694e2b1e3bef8dc0"
server_key_iv: "86c47ded8d8891810b8c8144efbc5257"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.322222 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:54.336252 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:54.338682 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:54.338897 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:54.339195 15863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:54.339527 15863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:54.339582 15863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.339624 15863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:54.339650 15863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.346235 15863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36863
I20260925 12:14:54.346276 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36863 every 8 connection(s)
I20260925 12:14:54.346591 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:14:54.349419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15863
I20260925 12:14:54.349499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:14:54.349697 392 external_mini_cluster.cc:1428] Setting key 1ed0a40f60e8db9f4364013411c5a7ea
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:54.357189 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.347296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59253 (local address 127.0.98.62:43811)
0925 12:14:54.347351 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.347353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.347612 (+ 259us) server_negotiation.cc:407] Connection header received
0925 12:14:54.348077 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.348078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.348088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.348144 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:54.349247 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.349928 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.351583 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.351790 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.351876 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.354155 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:54.354157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:54.354157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:54.354174 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:54.355623 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.356055 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.356056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.356057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.356128 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.356404 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.356405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.356405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.356592 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:54.356697 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.356998 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.357087 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:14:54.357373 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.347298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:59253)
0925 12:14:54.347480 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.347517 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.347989 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.348262 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.348265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.348943 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:54.349138 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.349143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.350001 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.350002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.351492 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.351496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.351884 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.351886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.352018 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.352604 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:54.352612 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:54.354053 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:54.355735 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.355742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.355745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.355976 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.356205 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.356206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.356206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.356329 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.356687 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:54.356695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:54.356854 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.356964 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.357129 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:14:54.357887 15992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:14:54.357998 15992 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:54.358199 15992 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:14:54.358729 15800 ts_manager.cc:194] Registered new tserver with Master: be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863)
I20260925 12:14:54.359368 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:14:54.376078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:54.459395 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:54.459618 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:54.459645 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:54.460904 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:54.460952 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:54.460981 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:14:54.462469 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:54.462667 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:54.462849 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:54.467988 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.468014 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.467996 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:54.468413 15997 server_base.cc:1034] running on GCE node
I20260925 12:14:54.468672 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:54.468938 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:54.470080 15997 hybrid_clock.cc:648] HybridClock initialized: now 1790338494470060 us; error 40 us; skew 500 ppm
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:54.472644 15997 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:14:54.473119 15997 webserver.cc:460] Webserver started at http://127.0.98.2:38595/ using document root <none> and password file <none>
I20260925 12:14:54.473361 15997 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:54.473429 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:54.473589 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:54.474686 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "09f70ab4a20af37e8e6422b444ff16c8"
server_key_iv: "846f57c898679fd70aeb5015417f5258"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.475069 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "09f70ab4a20af37e8e6422b444ff16c8"
server_key_iv: "846f57c898679fd70aeb5015417f5258"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.476470 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:14:54.477334 16012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.477512 15997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:14:54.477614 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "09f70ab4a20af37e8e6422b444ff16c8"
server_key_iv: "846f57c898679fd70aeb5015417f5258"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.477702 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:54.505447 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:54.507675 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:54.507861 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:54.508124 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:54.508446 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:54.508520 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.508582 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:54.508630 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.514750 15997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38815
I20260925 12:14:54.514807 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38815 every 8 connection(s)
I20260925 12:14:54.515162 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:14:54.521366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 15997
I20260925 12:14:54.521476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:14:54.521689 392 external_mini_cluster.cc:1428] Setting key 23dd209e8820d954a44e089e6ed53ce2
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:54.525585 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.516121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55727 (local address 127.0.98.62:43811)
0925 12:14:54.516192 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.516195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.516268 (+ 73us) server_negotiation.cc:407] Connection header received
0925 12:14:54.516916 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.516918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.516930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.516982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:54.517973 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.518493 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.519518 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.519736 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.519812 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.522414 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:54.522415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:54.522416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:54.522425 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:54.523905 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.524421 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.524422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.524422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.524473 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.524872 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.524872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.524873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.525043 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:54.525096 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.525455 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.525521 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 12:14:54.525942 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.515931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:55727)
0925 12:14:54.516108 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.516171 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.516802 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.517057 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.517061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.517644 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:54.517852 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.517857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.518598 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.518599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.519423 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.519425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.519830 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.519832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.519952 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.520567 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:54.520576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:54.522301 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:54.523989 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.523996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.523999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.524338 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.524617 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.524618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.524619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.524770 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.525123 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:54.525129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:54.525315 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.525436 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.525620 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260925 12:14:54.526460 16126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:14:54.526574 16126 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:54.526763 16126 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:14:54.527211 15800 ts_manager.cc:194] Registered new tserver with Master: b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
I20260925 12:14:54.527767 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55727
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:14:54.548429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:14:54.629698 16131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:14:54.629902 16131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:14:54.629925 16131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:14:54.631083 16131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:14:54.631126 16131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:14:54.631152 16131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:14:54.632601 16131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:14:54.632826 16131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:14:54.633035 16131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:14:54.638090 16136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:14:54.638165 16137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:54.638281 16131 server_base.cc:1034] running on GCE node
W20260925 12:14:54.638062 16139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:14:54.638527 16131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:14:54.638736 16131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:14:54.639875 16131 hybrid_clock.cc:648] HybridClock initialized: now 1790338494639851 us; error 38 us; skew 500 ppm
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:14:54.642628 16131 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:14:54.643211 16131 webserver.cc:460] Webserver started at http://127.0.98.3:46717/ using document root <none> and password file <none>
I20260925 12:14:54.643456 16131 fs_manager.cc:359] Metadata directory not provided
I20260925 12:14:54.643524 16131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:14:54.643673 16131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:14:54.644805 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5f020b3cfbc460f8e7debe76101110f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "8fceaec4f10fa279454e0852d03643ce"
server_key_iv: "0dc0c01f15f635287f1897c21e9b05c3"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.645160 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5f020b3cfbc460f8e7debe76101110f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "8fceaec4f10fa279454e0852d03643ce"
server_key_iv: "0dc0c01f15f635287f1897c21e9b05c3"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.646459 16131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:14:54.647217 16146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.647387 16131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:14:54.647491 16131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "d5f020b3cfbc460f8e7debe76101110f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "8fceaec4f10fa279454e0852d03643ce"
server_key_iv: "0dc0c01f15f635287f1897c21e9b05c3"
server_key_version: "encryptionkey@0"
I20260925 12:14:54.647563 16131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:14:54.666251 16131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:14:54.668434 16131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:14:54.668622 16131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:14:54.668869 16131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:14:54.669229 16131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:14:54.669314 16131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.669376 16131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:14:54.669407 16131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:14:54.675745 16131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260925 12:14:54.675789 16259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260925 12:14:54.676134 16131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 12:14:54 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:14:54.683625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16131
I20260925 12:14:54.683776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:14:54.683984 392 external_mini_cluster.cc:1428] Setting key a5e484eedb2588536f642278fa1c69e4
I20260925 12:14:54.686053 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.676924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41231 (local address 127.0.98.62:43811)
0925 12:14:54.676972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.676974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.677162 (+ 188us) server_negotiation.cc:407] Connection header received
0925 12:14:54.677701 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.677702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.677715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.677768 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:54.678810 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.679319 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.680354 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.680644 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.680715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.682988 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:54.682990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:54.682990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:54.682998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:54.684616 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.685137 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.685138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.685138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.685189 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.685411 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.685412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.685412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.685515 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:54.685572 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.685906 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.685985 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:14:54.686182 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.676882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:41231)
0925 12:14:54.677020 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.677067 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.677590 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.677851 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.677857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.678477 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:14:54.678696 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.678702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.679416 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.679417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.680247 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.680249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.680735 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.680736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.680856 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.681423 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:14:54.681432 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:14:54.682886 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:14:54.684737 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.684745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.684747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.685068 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.685255 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:14:54.685255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:14:54.685255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:14:54.685357 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:14:54.685603 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:14:54.685609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:14:54.685742 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.685843 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.685974 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260925 12:14:54.686699 16260 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:14:54.686785 16260 heartbeater.cc:464] Registering TS with master...
I20260925 12:14:54.686971 16260 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:14:54.687425 15800 ts_manager.cc:194] Registered new tserver with Master: d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005)
I20260925 12:14:54.687839 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41231
I20260925 12:14:54.695996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:14:54.710116 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.697456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.98.62:43811)
0925 12:14:54.697528 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.697530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.697689 (+ 159us) server_negotiation.cc:407] Connection header received
0925 12:14:54.697746 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.697747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.697756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.697796 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:14:54.698338 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.698839 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.699601 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.699787 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.699842 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.707343 (+ 7501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:14:54.707344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:14:54.707345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:14:54.707353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:14:54.708882 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.709282 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.709282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.709283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.709332 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:14:54.709568 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:14:54.709568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:14:54.709568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:14:54.709721 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:14:54.709795 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.709961 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.710025 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 12:14:54.711915 15800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:14:54.713616 15800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:14:54.720594 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.98.3:45005)
0925 12:14:54.716560 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.716566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.716620 (+ 54us) server_negotiation.cc:407] Connection header received
0925 12:14:54.716729 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.716731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.716750 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.716878 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:54.717064 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718194 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.719585 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.720190 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.720264 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.720322 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.720422 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.720460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260925 12:14:54.720603 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.0.1:45336)
0925 12:14:54.716557 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.716567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.716661 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.716888 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.716891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.716894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:54.716978 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.716981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718317 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.718318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.719495 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.719498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.720275 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.720275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.720336 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.720348 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.720409 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.720451 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":85,"thread_start_us":81,"threads_started":1}
I20260925 12:14:54.721104 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36863 (local address 127.0.0.1:59374)
0925 12:14:54.716198 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.716247 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.716338 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.716992 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.716992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.716995 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:54.717067 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.717070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718228 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.718229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.719879 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.719882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.720895 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.720896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.720959 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.720963 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.720991 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.721030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":100,"thread_start_us":63,"threads_started":1}
I20260925 12:14:54.721522 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.98.1:36863)
0925 12:14:54.716800 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.716806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.716817 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:14:54.716859 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.716861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.716878 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.716986 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:54.717169 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718106 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.719977 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.720790 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.721232 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.721302 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.721328 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.721412 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":108,"threads_started":1}
I20260925 12:14:54.721845 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38815 (local address 127.0.0.1:54904)
0925 12:14:54.717149 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.717160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.717214 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.717411 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.717412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.717414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:54.717451 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.717453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718656 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.718657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.719882 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.719884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.721500 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.721502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.721611 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.721617 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.721684 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.721737 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260925 12:14:54.721910 16194 tablet_service.cc:1467] Processing CreateTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 (DEFAULT_TABLE table=test-table [id=811e5d947fda49e4a3aa690a96a10a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:54.722132 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.716636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.0.98.2:38815)
0925 12:14:54.716833 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.716837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.717207 (+ 370us) server_negotiation.cc:407] Connection header received
0925 12:14:54.717294 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.717296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.717310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.717392 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:54.717524 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.718547 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.720681 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.721301 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.721400 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.721459 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.721954 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.722018 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260925 12:14:54.722334 16194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f0ea5f5f04e269062bcb948f1faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:54.725195 16060 tablet_service.cc:1467] Processing CreateTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 (DEFAULT_TABLE table=test-table [id=811e5d947fda49e4a3aa690a96a10a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:54.725597 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f0ea5f5f04e269062bcb948f1faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:54.728091 16278 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Bootstrap starting.
I20260925 12:14:54.728106 16277 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Bootstrap starting.
I20260925 12:14:54.728431 15926 tablet_service.cc:1467] Processing CreateTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 (DEFAULT_TABLE table=test-table [id=811e5d947fda49e4a3aa690a96a10a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:14:54.728813 16278 tablet_bootstrap.cc:654] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:54.728839 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316f0ea5f5f04e269062bcb948f1faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:14:54.728909 16277 tablet_bootstrap.cc:654] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:54.729235 16278 log.cc:824] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:54.729573 16277 log.cc:824] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:54.729969 16278 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: No bootstrap required, opened a new log
I20260925 12:14:54.730048 16278 ts_tablet_manager.cc:1397] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:14:54.730438 16277 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: No bootstrap required, opened a new log
I20260925 12:14:54.730526 16277 ts_tablet_manager.cc:1397] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:14:54.731554 16278 raft_consensus.cc:348] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.731732 16278 raft_consensus.cc:374] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:54.731796 16278 raft_consensus.cc:729] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d3292fbde44144abf28f7f3e7c76a9, State: Initialized, Role: FOLLOWER
I20260925 12:14:54.731959 16278 consensus_queue.cc:260] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.732472 16126 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
I20260925 12:14:54.732765 16282 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Bootstrap starting.
I20260925 12:14:54.732471 16278 ts_tablet_manager.cc:1428] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:54.732980 16277 raft_consensus.cc:348] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.733186 16277 raft_consensus.cc:374] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:54.733276 16277 raft_consensus.cc:729] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f020b3cfbc460f8e7debe76101110f, State: Initialized, Role: FOLLOWER
I20260925 12:14:54.733433 16277 consensus_queue.cc:260] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.733501 16282 tablet_bootstrap.cc:654] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Neither blocks nor log segments found. Creating new log.
I20260925 12:14:54.733774 16277 ts_tablet_manager.cc:1428] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 12:14:54.733945 16260 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
I20260925 12:14:54.734143 16282 log.cc:824] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Log is configured to *not* fsync() on all Append() calls
I20260925 12:14:54.734983 16282 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: No bootstrap required, opened a new log
I20260925 12:14:54.735046 16282 ts_tablet_manager.cc:1397] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:54.736584 16282 raft_consensus.cc:348] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.736740 16282 raft_consensus.cc:374] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:14:54.736763 16282 raft_consensus.cc:729] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01383017b64a0e9ffbd37159671917, State: Initialized, Role: FOLLOWER
I20260925 12:14:54.736886 16282 consensus_queue.cc:260] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.737150 16282 ts_tablet_manager.cc:1428] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:14:54.737250 15992 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
W20260925 12:14:54.765816 16127 tablet.cc:2369] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:14:54.847282 15993 tablet.cc:2369] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:14:54.883556 16285 raft_consensus.cc:482] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:14:54.883694 16285 raft_consensus.cc:504] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.884110 16285 leader_election.cc:288] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
I20260925 12:14:54.888651 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.884579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53879 (local address 127.0.98.3:45005)
0925 12:14:54.884638 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.884640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.884778 (+ 138us) server_negotiation.cc:407] Connection header received
0925 12:14:54.884858 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.884860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.884874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.884932 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:54.885191 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.885958 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.887624 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.888222 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.888300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.888310 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.888536 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.888581 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:14:54.888667 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.884461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.98.1:53879)
0925 12:14:54.884686 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.884698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.884770 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.884959 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.884961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.884963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:54.885091 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.885094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.886085 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.886086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.887514 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.887517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.888333 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.888334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.888424 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.888431 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.888516 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.888565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":109,"thread_start_us":139,"threads_started":1}
I20260925 12:14:54.888758 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.884838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36573 (local address 127.0.98.2:38815)
0925 12:14:54.884889 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.884892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.884984 (+ 92us) server_negotiation.cc:407] Connection header received
0925 12:14:54.885039 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.885040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.885050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.885091 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:14:54.885258 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.885966 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.887651 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.888222 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.888300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.888310 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.888631 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.888681 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:14:54.889025 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.884739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38815 (local address 127.0.98.1:36573)
0925 12:14:54.884865 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.884876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:14:54.884975 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:14:54.885121 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:14:54.885121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:14:54.885123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:14:54.885186 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.885188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.886085 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.886086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.887514 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:14:54.887517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.888332 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.888334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:14:54.888424 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.888433 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 12:14:54.888603 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 12:14:54.888636 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":84,"thread_start_us":48,"threads_started":1}
I20260925 12:14:54.889129 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f0ea5f5f04e269062bcb948f1faf9" candidate_uuid: "be01383017b64a0e9ffbd37159671917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f020b3cfbc460f8e7debe76101110f" is_pre_election: true
I20260925 12:14:54.889314 16214 raft_consensus.cc:2454] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01383017b64a0e9ffbd37159671917 in term 0.
I20260925 12:14:54.889436 16080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f0ea5f5f04e269062bcb948f1faf9" candidate_uuid: "be01383017b64a0e9ffbd37159671917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" is_pre_election: true
I20260925 12:14:54.889602 16080 raft_consensus.cc:2454] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01383017b64a0e9ffbd37159671917 in term 0.
I20260925 12:14:54.889766 15879 leader_election.cc:302] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01383017b64a0e9ffbd37159671917, d5f020b3cfbc460f8e7debe76101110f; no voters:
I20260925 12:14:54.889950 16285 raft_consensus.cc:2781] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:14:54.890025 16285 raft_consensus.cc:482] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:14:54.890058 16285 raft_consensus.cc:3037] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:54.890833 16285 raft_consensus.cc:504] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.890981 16285 leader_election.cc:288] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 1 election: Requested vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
I20260925 12:14:54.891188 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f0ea5f5f04e269062bcb948f1faf9" candidate_uuid: "be01383017b64a0e9ffbd37159671917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f020b3cfbc460f8e7debe76101110f"
I20260925 12:14:54.891258 16080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316f0ea5f5f04e269062bcb948f1faf9" candidate_uuid: "be01383017b64a0e9ffbd37159671917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
I20260925 12:14:54.891288 16214 raft_consensus.cc:3037] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:54.891367 16080 raft_consensus.cc:3037] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:14:54.892076 16214 raft_consensus.cc:2454] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01383017b64a0e9ffbd37159671917 in term 1.
I20260925 12:14:54.892098 16080 raft_consensus.cc:2454] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01383017b64a0e9ffbd37159671917 in term 1.
I20260925 12:14:54.892267 15879 leader_election.cc:302] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01383017b64a0e9ffbd37159671917, d5f020b3cfbc460f8e7debe76101110f; no voters:
I20260925 12:14:54.892373 16285 raft_consensus.cc:2781] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:14:54.892561 16285 raft_consensus.cc:686] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 LEADER]: Becoming Leader. State: Replica: be01383017b64a0e9ffbd37159671917, State: Running, Role: LEADER
I20260925 12:14:54.892704 16285 consensus_queue.cc:237] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:14:54.893585 15800 catalog_manager.cc:5495] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 reported cstate change: term changed from 0 to 1, leader changed from <none> to be01383017b64a0e9ffbd37159671917 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "be01383017b64a0e9ffbd37159671917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } health_report { overall_health: UNKNOWN } } }
I20260925 12:14:54.906633 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.904015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.0.98.1:36863)
0925 12:14:54.904126 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 12:14:54.904128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:14:54.904136 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:14:54.904212 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:14:54.904213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:14:54.904237 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:14:54.904338 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:14:54.904474 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.905002 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.905810 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:14:54.906026 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:14:54.906107 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:14:54.906292 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:14:54.906407 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:14:54.906503 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 12:14:54.906523 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 12:14:54.906543 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 12:14:54.908622 16080 raft_consensus.cc:1261] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Refusing update from remote peer be01383017b64a0e9ffbd37159671917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:54.908631 16214 raft_consensus.cc:1261] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Refusing update from remote peer be01383017b64a0e9ffbd37159671917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:14:54.908982 16285 consensus_queue.cc:1035] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:54.909157 16288 consensus_queue.cc:1035] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:14:54.911652 16292 mvcc.cc:205] Tried to move back new op lower bound from 7333226475142660096 to 7333226475081121792. Current Snapshot: MvccSnapshot[applied={T|T < 7333226475142660096}]
I20260925 12:14:54.912671 16293 mvcc.cc:205] Tried to move back new op lower bound from 7333226475142660096 to 7333226475081121792. Current Snapshot: MvccSnapshot[applied={T|T < 7333226475142660096}]
I20260925 12:14:54.913877 15800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:54.913996 15800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:14:54.914741 15800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Sending DeleteTablet for 3 replicas of tablet 316f0ea5f5f04e269062bcb948f1faf9
I20260925 12:14:54.915045 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59374
I20260925 12:14:54.915153 16060 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54904
I20260925 12:14:54.915200 16194 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260925 12:14:54.915259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15863
I20260925 12:14:54.915838 16302 tablet_replica.cc:330] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: stopping tablet replica
I20260925 12:14:54.915934 16302 raft_consensus.cc:2229] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:54.916033 16302 raft_consensus.cc:2258] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:54.916524 16302 ts_tablet_manager.cc:1905] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:54.916947 16301 tablet_replica.cc:330] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: stopping tablet replica
I20260925 12:14:54.917064 16301 raft_consensus.cc:2229] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:14:54.917184 16301 raft_consensus.cc:2258] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:14:54.917588 16301 ts_tablet_manager.cc:1905] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:14:54.918558 16302 ts_tablet_manager.cc:1918] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:54.918635 16302 log.cc:1196] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/316f0ea5f5f04e269062bcb948f1faf9
I20260925 12:14:54.918920 16302 ts_tablet_manager.cc:1939] T 316f0ea5f5f04e269062bcb948f1faf9 P d5f020b3cfbc460f8e7debe76101110f: Deleting consensus metadata
I20260925 12:14:54.919018 16301 ts_tablet_manager.cc:1918] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:14:54.919096 16301 log.cc:1196] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/316f0ea5f5f04e269062bcb948f1faf9
I20260925 12:14:54.919373 16301 ts_tablet_manager.cc:1939] T 316f0ea5f5f04e269062bcb948f1faf9 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting consensus metadata
I20260925 12:14:54.919770 15784 catalog_manager.cc:4841] TS b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): tablet 316f0ea5f5f04e269062bcb948f1faf9 (table test-table [id=811e5d947fda49e4a3aa690a96a10a09]) successfully deleted
I20260925 12:14:54.920109 15784 catalog_manager.cc:4841] TS d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): tablet 316f0ea5f5f04e269062bcb948f1faf9 (table test-table [id=811e5d947fda49e4a3aa690a96a10a09]) successfully deleted
W20260925 12:14:54.921748 15784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36863 (error 108)
I20260925 12:14:54.921994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15997
I20260925 12:14:54.922804 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:14:54.922654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36863 (local address 127.0.0.1:59398)
0925 12:14:54.922718 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 12:14:54.922762 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 12:14:54.922941 15784 catalog_manager.cc:4584] TS be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 316f0ea5f5f04e269062bcb948f1faf9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36863: connect: Connection refused (error 111)
I20260925 12:14:54.927722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16131
I20260925 12:14:54.932099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 15768
I20260925 12:15:14.937516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--webserver_interface=127.0.98.62
--webserver_port=42431
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:15.019747 16304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:15.020016 16304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:15.020064 16304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:15.021250 16304 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:15:15.021335 16304 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:15:15.021359 16304 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:15:15.021378 16304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:15.021410 16304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:15.021441 16304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:15.021472 16304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:15.021502 16304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:15.022997 16304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:15.023291 16304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:15.023528 16304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:15:15.032702 16304 server_base.cc:1034] running on GCE node
W20260925 12:15:15.032709 16311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.032717 16310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.032709 16313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:15.033277 16304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:15.033633 16304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:15.034821 16304 hybrid_clock.cc:648] HybridClock initialized: now 1790338515034803 us; error 49 us; skew 500 ppm
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:15.044729 16304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:15:15.045473 16304 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260925 12:15:15.045766 16304 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:15.045856 16304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:15.047411 16304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.048326 16320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.048691 16304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.048806 16304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccf5fd333c0b49f1b34974882c96204f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "dad1f6a77dc808721c3f2733091bf3f6"
server_key_iv: "3b604745c8ae4ef7a5bf9b8e608e9a35"
server_key_version: "encryptionkey@0"
I20260925 12:15:15.049275 16304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:15.061589 16304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:15.063534 16304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:15.063719 16304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:15.067883 16304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43811
I20260925 12:15:15.067916 16372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43811 every 8 connection(s)
I20260925 12:15:15.068316 16304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:15.069886 16373 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:15:15.070591 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap starting.
I20260925 12:15:15.073241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16304
I20260925 12:15:15.073586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36863
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=38397
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:15:15.079672 16373 log.cc:824] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:15.081728 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:15:15.082026 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap complete.
I20260925 12:15:15.084482 16373 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:15.084841 16373 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Initialized, Role: FOLLOWER
I20260925 12:15:15.085026 16373 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:15.085115 16373 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:15.085175 16373 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:15.085223 16373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 1 FOLLOWER]: Advancing to term 2
I20260925 12:15:15.086257 16373 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:15.086385 16373 leader_election.cc:302] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf5fd333c0b49f1b34974882c96204f; no voters:
I20260925 12:15:15.086591 16373 leader_election.cc:288] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 12:15:15.086627 16377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 12:15:15.086737 16377 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 LEADER]: Becoming Leader. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Running, Role: LEADER
I20260925 12:15:15.086870 16373 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:15.086885 16377 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:15.087272 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:15:15.087285 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf5fd333c0b49f1b34974882c96204f. Latest consensus state: current_term: 2 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:15:15.087386 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:15.087411 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:15.087627 16384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:15.088570 16384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:15.088721 16384 catalog_manager.cc:1259] Loaded cluster ID: 30115476a8f74610bf251db35d828cb7
I20260925 12:15:15.088945 16384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:15:15.090113 16384 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:15.090297 16384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Loaded TSK: 0
I20260925 12:15:15.099793 16384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Generated new TSK 1
I20260925 12:15:15.099905 16384 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 12:15:15.159595 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:15.159816 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:15.159840 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:15.161125 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:15.161172 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:15.161201 16375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:15.162705 16375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38397
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:15.162977 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:15.163225 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:15.168819 16400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.168864 16401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.168844 16403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:15.169171 16375 server_base.cc:1034] running on GCE node
I20260925 12:15:15.169400 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:15.169674 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:15.170861 16375 hybrid_clock.cc:648] HybridClock initialized: now 1790338515170848 us; error 27 us; skew 500 ppm
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:15.173686 16375 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:15:15.174220 16375 webserver.cc:460] Webserver started at http://127.0.98.1:38397/ using document root <none> and password file <none>
I20260925 12:15:15.174463 16375 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:15.174525 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:15.175949 16375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.176707 16410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.176882 16375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:15.176988 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "be01383017b64a0e9ffbd37159671917"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "34fa8e254ac2f1b5694e2b1e3bef8dc0"
server_key_iv: "86c47ded8d8891810b8c8144efbc5257"
server_key_version: "encryptionkey@0"
I20260925 12:15:15.177346 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:15.186604 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:15.189000 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:15.189167 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:15.189415 16375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:15.189896 16417 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:15:15.191329 16375 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:15:15.191383 16375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:15:15.191416 16375 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:15:15.192096 16375 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:15:15.192142 16375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:15:15.192188 16417 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Bootstrap starting.
I20260925 12:15:15.200063 16375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36863
I20260925 12:15:15.200595 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:15.203570 16417 log.cc:824] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:15.203902 16524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36863 every 8 connection(s)
I20260925 12:15:15.206378 16417 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:15:15.206920 16417 tablet_bootstrap.cc:492] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Bootstrap complete.
I20260925 12:15:15.208158 16417 ts_tablet_manager.cc:1397] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:15.209578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16375
I20260925 12:15:15.209848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38815
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=38595
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:15:15.211413 16417 raft_consensus.cc:348] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:15:15.211815 16417 raft_consensus.cc:729] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01383017b64a0e9ffbd37159671917, State: Initialized, Role: FOLLOWER
I20260925 12:15:15.212016 16417 consensus_queue.cc:260] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } } peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } }
I20260925 12:15:15.212342 16417 ts_tablet_manager.cc:1428] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 12:15:15.213344 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.201620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59971 (local address 127.0.98.62:43811)
0925 12:15:15.202701 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.202710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.202745 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:15:15.202876 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.202882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.202935 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.203068 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:15.203997 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.205032 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.206085 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.206624 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.206747 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.209108 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:15.209112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:15.209112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:15.209124 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:15.211149 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.211715 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.211719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.211719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.211849 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.212156 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.212157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.212158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.212484 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:15.212604 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.213008 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.213152 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":77,"threads_started":1}
I20260925 12:15:15.213378 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.201283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:59971)
0925 12:15:15.201811 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.201858 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.202471 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.203205 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.203209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.203585 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:15.203850 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.203857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.205180 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.205181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.205977 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.205979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.206737 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.206738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.206881 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.207463 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:15.207478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:15.208986 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:15.211267 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.211275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.211277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.211611 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.211930 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.211931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.211932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.212065 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.212593 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:15.212599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:15.212764 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.212926 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.213131 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260925 12:15:15.214619 16525 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:15:15.214711 16525 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:15.214877 16525 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:15:15.215759 16337 ts_manager.cc:194] Registered new tserver with Master: be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863)
I20260925 12:15:15.217100 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59971
I20260925 12:15:15.218258 16525 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
I20260925 12:15:15.224772 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.216709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.0.98.1:36863)
0925 12:15:15.220646 (+ 3937us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.220652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.220663 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:15.220700 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.220703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.220719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.220813 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:15.221024 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.222019 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.223547 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.224308 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.224377 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.224428 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.224654 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.224696 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3892,"thread_start_us":76,"threads_started":1}
I20260925 12:15:15.224771 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.216438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36863 (local address 127.0.0.1:36732)
0925 12:15:15.216883 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.216916 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.216979 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.220843 (+ 3864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.220847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.220850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:15.220928 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.220930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.222114 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.222115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.223445 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.223449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.224420 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.224421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.224529 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.224553 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.224630 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.224685 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":134,"threads_started":1}
I20260925 12:15:15.225421 16459 tablet_service.cc:1514] Processing DeleteTablet for tablet 316f0ea5f5f04e269062bcb948f1faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:14:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36732
I20260925 12:15:15.226054 16534 tablet_replica.cc:330] stopping tablet replica
I20260925 12:15:15.226186 16534 raft_consensus.cc:2229] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:15.226253 16534 raft_consensus.cc:2258] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:15.226666 16534 ts_tablet_manager.cc:1905] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:15.229108 16534 ts_tablet_manager.cc:1918] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:15.229179 16534 log.cc:1196] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal/wals/316f0ea5f5f04e269062bcb948f1faf9
I20260925 12:15:15.229393 16534 ts_tablet_manager.cc:1939] T 316f0ea5f5f04e269062bcb948f1faf9 P be01383017b64a0e9ffbd37159671917: Deleting consensus metadata
I20260925 12:15:15.229960 16321 catalog_manager.cc:4841] TS be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863): tablet 316f0ea5f5f04e269062bcb948f1faf9 (table test-table [id=811e5d947fda49e4a3aa690a96a10a09]) successfully deleted
W20260925 12:15:15.296447 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:15.296818 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:15.296859 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:15.298149 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:15.298216 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:15.298260 16530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:15.299767 16530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38595
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:15.300042 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:15.300288 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:15.305600 16540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:15.305665 16530 server_base.cc:1034] running on GCE node
W20260925 12:15:15.305745 16539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.305856 16542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:15.306077 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:15.306313 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:15.307480 16530 hybrid_clock.cc:648] HybridClock initialized: now 1790338515307475 us; error 33 us; skew 500 ppm
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:15.309926 16530 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:15:15.310355 16530 webserver.cc:460] Webserver started at http://127.0.98.2:38595/ using document root <none> and password file <none>
I20260925 12:15:15.310609 16530 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:15.310699 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:15.312070 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.312743 16549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.312973 16530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.313087 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "09f70ab4a20af37e8e6422b444ff16c8"
server_key_iv: "846f57c898679fd70aeb5015417f5258"
server_key_version: "encryptionkey@0"
I20260925 12:15:15.313508 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:15.338858 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:15.340885 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:15.341060 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:15.341306 16530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:15.341617 16530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:15.341675 16530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.341718 16530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:15.341743 16530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.347488 16530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38815
I20260925 12:15:15.347532 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38815 every 8 connection(s)
I20260925 12:15:15.347879 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:15.356421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16530
I20260925 12:15:15.356827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45005
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=46717
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:15:15.357801 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.348753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50419 (local address 127.0.98.62:43811)
0925 12:15:15.348796 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.348798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.349003 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:15:15.349607 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.349609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.349621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.349699 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:15.350729 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.351230 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.352308 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.352619 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.352708 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.354886 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:15.354888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:15.354888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:15.354895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:15.356239 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.356793 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.356794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.356794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.356849 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.357165 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.357166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.357166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.357279 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:15.357333 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.357664 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.357736 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:15:15.358126 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.348721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:50419)
0925 12:15:15.348862 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.348908 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.349431 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.349790 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.349797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.350412 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:15.350610 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.350615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.351331 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.351332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.352210 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.352212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.352721 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.352722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.352857 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.353339 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:15.353348 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:15.354735 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:15.356331 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.356339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.356341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.356703 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.356953 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.356954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.356955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.357095 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.357348 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:15.357354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:15.357525 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.357643 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.357821 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260925 12:15:15.358728 16663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:15:15.358847 16663 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:15.359052 16663 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:15:15.359473 16337 ts_manager.cc:194] Registered new tserver with Master: b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
I20260925 12:15:15.359809 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50419
W20260925 12:15:15.440001 16666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:15.440227 16666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:15.440250 16666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:15.441473 16666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:15.441521 16666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:15.441550 16666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:15.443075 16666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46717
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:15.443265 16666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:15.443446 16666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:15.448742 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.448686 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:15.448686 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:15.448737 16666 server_base.cc:1034] running on GCE node
I20260925 12:15:15.449126 16666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:15.449378 16666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:15.450568 16666 hybrid_clock.cc:648] HybridClock initialized: now 1790338515450582 us; error 70 us; skew 500 ppm
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:15.453295 16666 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:15:15.453816 16666 webserver.cc:460] Webserver started at http://127.0.98.3:46717/ using document root <none> and password file <none>
I20260925 12:15:15.454052 16666 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:15.454140 16666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:15.455549 16666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:15.456173 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.456336 16666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:15.456441 16666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "d5f020b3cfbc460f8e7debe76101110f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "8fceaec4f10fa279454e0852d03643ce"
server_key_iv: "0dc0c01f15f635287f1897c21e9b05c3"
server_key_version: "encryptionkey@0"
I20260925 12:15:15.456866 16666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:15.471369 16666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:15.473680 16666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:15.473868 16666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:15.474124 16666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:15.474452 16666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:15.474519 16666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.474562 16666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:15.474607 16666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:15.480640 16666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260925 12:15:15.480688 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260925 12:15:15.481029 16666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:15.483210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16666
Sep 25 12:15:15 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:15.491431 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.483618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.98.62:43811)
0925 12:15:15.483733 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.483735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.483893 (+ 158us) server_negotiation.cc:407] Connection header received
0925 12:15:15.483956 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.483956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.483966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.484059 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:15.484812 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.485553 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.486581 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.486836 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.486909 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.488458 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:15.488459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:15.488460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:15.488467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:15.490210 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.490605 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.490607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.490607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.490668 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.490879 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.490880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.490880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.490999 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:15.491193 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.491230 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.491308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260925 12:15:15.491940 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.481742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47267 (local address 127.0.98.62:43811)
0925 12:15:15.481813 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.481815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.482066 (+ 251us) server_negotiation.cc:407] Connection header received
0925 12:15:15.482565 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.482566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.482576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.482630 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:15.483649 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.484436 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.485846 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.486135 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.486217 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.488648 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:15.488649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:15.488650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:15.488657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:15.490325 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.490827 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.490829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.490829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.490888 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.491162 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.491163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.491163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.491271 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:15.491348 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.491742 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.491847 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:15:15.492069 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.481737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:47267)
0925 12:15:15.481923 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.481976 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.482445 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.482683 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.482686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.483291 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:15.483549 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.483555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.484587 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.484588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.485755 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.485758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.486243 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.486244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.486366 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.486904 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:15.486915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:15.488541 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:15.490506 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.490513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.490516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.490742 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.490977 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:15.490978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:15.490978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:15.491089 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:15.491346 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:15.491353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:15.491530 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.491652 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.491823 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260925 12:15:15.492604 16795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43811
I20260925 12:15:15.492725 16795 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:15.492911 16795 heartbeater.cc:511] Master 127.0.98.62:43811 requested a full tablet report, sending...
I20260925 12:15:15.493330 16337 ts_manager.cc:194] Registered new tserver with Master: d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005)
I20260925 12:15:15.493721 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47267
I20260925 12:15:15.503263 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:15.505975 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.503655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.0.98.62:43811)
0925 12:15:15.503732 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.503734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.503819 (+ 85us) server_negotiation.cc:407] Connection header received
0925 12:15:15.503871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.503871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.503907 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.504006 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:15.504098 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.504638 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.505375 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.505562 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.505617 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.505756 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:15.505802 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 12:15:15.505926 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40086: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 12:15:15.506064 16528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40086: token expired
W20260925 12:15:15.506172 16264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43811)
I20260925 12:15:15.512612 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.506433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.98.62:43811)
0925 12:15:15.506495 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.506497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.506506 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:15.506565 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.506566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.506575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.506618 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:15.507122 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.507686 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.508389 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.508684 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.508772 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.509932 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:15.509933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:15.509934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:15.509941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:15.511361 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.511756 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.511757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.511757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.511813 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:15.512029 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:15.512030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:15.512030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:15.512147 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:15.512294 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.512398 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.512480 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 12:15:15.513854 16337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:15.515463 16337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:15.518096 16459 tablet_service.cc:1467] Processing CreateTablet for tablet 88b15574c3304b4eba5165bd76104890 (DEFAULT_TABLE table=test-table [id=e106951648fc40058ec3d34e9be95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:15.518344 16459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b15574c3304b4eba5165bd76104890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:15.520000 16417 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Bootstrap starting.
I20260925 12:15:15.520313 16417 tablet_bootstrap.cc:654] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:15.521158 16417 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: No bootstrap required, opened a new log
I20260925 12:15:15.521214 16417 ts_tablet_manager.cc:1397] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 12:15:15.521330 16417 raft_consensus.cc:348] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.521406 16417 raft_consensus.cc:374] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:15.521456 16417 raft_consensus.cc:729] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01383017b64a0e9ffbd37159671917, State: Initialized, Role: FOLLOWER
I20260925 12:15:15.521507 16417 consensus_queue.cc:260] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.521593 16417 ts_tablet_manager.cc:1428] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:15.521607 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.517618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38815 (local address 127.0.0.1:34612)
0925 12:15:15.517799 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.517808 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.517856 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.518118 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.518119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.518121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:15.518176 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.518179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.519313 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.519314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.520284 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.520285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.521351 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.521352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.521416 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.521420 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.521481 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.521526 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
I20260925 12:15:15.521625 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.517735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.0.98.2:38815)
0925 12:15:15.517944 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.517951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.517965 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:15:15.518006 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.518007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.518019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.518121 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:15.518262 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.519220 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.520387 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.521268 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.521351 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.521415 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.521493 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.521543 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260925 12:15:15.521991 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.517517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.0.1:33602)
0925 12:15:15.517613 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.517627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.517684 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.518450 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.518451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.518453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:15.518494 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.518495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.519802 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.519803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.520895 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.520898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.521804 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.521805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.521860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.521863 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.521893 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.521936 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 12:15:15.522378 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.517663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.98.3:45005)
0925 12:15:15.518269 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.518274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.518283 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:15.518321 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.518324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.518338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.518438 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:15.518602 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.519694 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.520990 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.521702 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.522152 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.522215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.522249 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.522291 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":98,"threads_started":1}
I20260925 12:15:15.522365 16597 tablet_service.cc:1467] Processing CreateTablet for tablet 88b15574c3304b4eba5165bd76104890 (DEFAULT_TABLE table=test-table [id=e106951648fc40058ec3d34e9be95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:15.522743 16597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b15574c3304b4eba5165bd76104890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:15.523001 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 88b15574c3304b4eba5165bd76104890 (DEFAULT_TABLE table=test-table [id=e106951648fc40058ec3d34e9be95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:15.523372 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b15574c3304b4eba5165bd76104890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:15.525715 16805 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Bootstrap starting.
I20260925 12:15:15.526048 16806 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Bootstrap starting.
I20260925 12:15:15.526593 16805 tablet_bootstrap.cc:654] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:15.526677 16806 tablet_bootstrap.cc:654] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:15.527258 16805 log.cc:824] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:15.527258 16806 log.cc:824] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:15.528040 16805 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: No bootstrap required, opened a new log
I20260925 12:15:15.528124 16805 ts_tablet_manager.cc:1397] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:15.528192 16806 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: No bootstrap required, opened a new log
I20260925 12:15:15.528258 16806 ts_tablet_manager.cc:1397] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:15.530228 16806 raft_consensus.cc:348] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.530228 16805 raft_consensus.cc:348] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.530429 16805 raft_consensus.cc:374] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:15.530429 16806 raft_consensus.cc:374] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:15.530468 16806 raft_consensus.cc:729] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f020b3cfbc460f8e7debe76101110f, State: Initialized, Role: FOLLOWER
I20260925 12:15:15.530468 16805 raft_consensus.cc:729] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d3292fbde44144abf28f7f3e7c76a9, State: Initialized, Role: FOLLOWER
I20260925 12:15:15.530599 16805 consensus_queue.cc:260] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.530599 16806 consensus_queue.cc:260] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.530896 16805 ts_tablet_manager.cc:1428] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:15:15.530896 16806 ts_tablet_manager.cc:1428] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:15.530972 16795 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
I20260925 12:15:15.531085 16663 heartbeater.cc:503] Master 127.0.98.62:43811 was elected leader, sending a full tablet report...
W20260925 12:15:15.598508 16664 tablet.cc:2369] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:15.631172 16809 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:15.631297 16809 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.631664 16809 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863)
I20260925 12:15:15.634907 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.631893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.98.2:35673)
0925 12:15:15.631972 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.631987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.632060 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.632223 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.632224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.632226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:15.632309 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.632312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.633122 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.633123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.633987 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.633989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.634648 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.634734 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.634738 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.634789 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.634833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":42}
I20260925 12:15:15.634924 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.631986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35673 (local address 127.0.98.3:45005)
0925 12:15:15.632034 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.632036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.632047 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:15.632120 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.632121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.632133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.632190 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:15.632404 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.633033 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634091 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.634560 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634614 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.634623 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.634824 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.634866 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:15:15.635128 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.632100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59189 (local address 127.0.98.1:36863)
0925 12:15:15.632148 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.632151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.632349 (+ 198us) server_negotiation.cc:407] Connection header received
0925 12:15:15.632378 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.632379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.632392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.632448 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:15.632683 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.633215 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634285 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.634697 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634766 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.634778 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.634957 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.635046 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:15:15.635318 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b15574c3304b4eba5165bd76104890" candidate_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f020b3cfbc460f8e7debe76101110f" is_pre_election: true
I20260925 12:15:15.635388 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.632057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36863 (local address 127.0.98.2:59189)
0925 12:15:15.632240 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.632252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.632305 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.632524 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:15.632524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:15.632526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:15.632603 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.632604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.633321 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.633322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.634205 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:15.634207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.634777 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.634778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:15.634864 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.634868 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:15.635272 (+ 404us) client_negotiation.cc:240] Negotiation successful
0925 12:15:15.635318 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260925 12:15:15.635504 16749 raft_consensus.cc:2454] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d3292fbde44144abf28f7f3e7c76a9 in term 0.
I20260925 12:15:15.635803 16550 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f; no voters:
I20260925 12:15:15.635941 16809 raft_consensus.cc:2781] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:15.636008 16809 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:15.636030 16809 raft_consensus.cc:3037] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:15.635710 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b15574c3304b4eba5165bd76104890" candidate_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be01383017b64a0e9ffbd37159671917" is_pre_election: true
I20260925 12:15:15.636394 16479 raft_consensus.cc:2454] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d3292fbde44144abf28f7f3e7c76a9 in term 0.
I20260925 12:15:15.636948 16809 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.637106 16809 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [CANDIDATE]: Term 1 election: Requested vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863)
I20260925 12:15:15.637280 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b15574c3304b4eba5165bd76104890" candidate_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f020b3cfbc460f8e7debe76101110f"
I20260925 12:15:15.637317 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b15574c3304b4eba5165bd76104890" candidate_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be01383017b64a0e9ffbd37159671917"
I20260925 12:15:15.637362 16749 raft_consensus.cc:3037] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:15.637422 16479 raft_consensus.cc:3037] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:15.638130 16749 raft_consensus.cc:2454] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d3292fbde44144abf28f7f3e7c76a9 in term 1.
I20260925 12:15:15.638289 16550 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f; no voters:
I20260925 12:15:15.638305 16479 raft_consensus.cc:2454] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d3292fbde44144abf28f7f3e7c76a9 in term 1.
I20260925 12:15:15.638408 16809 raft_consensus.cc:2781] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:15.638562 16809 raft_consensus.cc:686] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 LEADER]: Becoming Leader. State: Replica: b8d3292fbde44144abf28f7f3e7c76a9, State: Running, Role: LEADER
I20260925 12:15:15.638675 16809 consensus_queue.cc:237] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:15.639508 16336 catalog_manager.cc:5495] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } health_report { overall_health: UNKNOWN } } }
I20260925 12:15:15.669590 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:15.666499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.0.98.2:38815)
0925 12:15:15.666595 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:15.666597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:15.666605 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:15.666671 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:15.666672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:15.666702 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:15.666778 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:15.667327 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.668071 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.668880 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:15.669091 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:15.669159 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:15.669287 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:15.669393 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:15.669454 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:15.669485 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 12:15:15.669504 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 12:15:15.671602 16749 raft_consensus.cc:1261] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Refusing update from remote peer b8d3292fbde44144abf28f7f3e7c76a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:15.671602 16479 raft_consensus.cc:1261] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Refusing update from remote peer b8d3292fbde44144abf28f7f3e7c76a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:15.671985 16812 consensus_queue.cc:1035] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:15.672174 16809 consensus_queue.cc:1035] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:15.676951 16335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:15.677076 16335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:15.677896 16335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Sending DeleteTablet for 3 replicas of tablet 88b15574c3304b4eba5165bd76104890
I20260925 12:15:15.678272 16459 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b15574c3304b4eba5165bd76104890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36732
I20260925 12:15:15.678261 16597 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b15574c3304b4eba5165bd76104890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34612
I20260925 12:15:15.678274 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b15574c3304b4eba5165bd76104890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33602
I20260925 12:15:15.678432 16534 tablet_replica.cc:330] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: stopping tablet replica
I20260925 12:15:15.678503 16534 raft_consensus.cc:2229] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:15.678553 16534 raft_consensus.cc:2258] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:15.678745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16375
I20260925 12:15:15.678925 16534 ts_tablet_manager.cc:1905] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:15.679801 16825 tablet_replica.cc:330] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: stopping tablet replica
I20260925 12:15:15.679924 16825 raft_consensus.cc:2229] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:15:15.680022 16825 raft_consensus.cc:2258] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:15.680366 16825 ts_tablet_manager.cc:1905] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:15.682318 16825 ts_tablet_manager.cc:1918] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:15.682500 16825 log.cc:1196] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/88b15574c3304b4eba5165bd76104890
I20260925 12:15:15.682796 16825 ts_tablet_manager.cc:1939] T 88b15574c3304b4eba5165bd76104890 P b8d3292fbde44144abf28f7f3e7c76a9: Deleting consensus metadata
I20260925 12:15:15.683223 16321 catalog_manager.cc:4841] TS b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): tablet 88b15574c3304b4eba5165bd76104890 (table test-table [id=e106951648fc40058ec3d34e9be95cba]) successfully deleted
W20260925 12:15:15.684195 16321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36863 (error 108)
W20260925 12:15:15.685353 16801 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:15:15.684782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36863 (local address 127.0.0.1:36738)
0925 12:15:15.684844 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:15.684855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:15.684897 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:15.685318 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 12:15:15.685559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16530
W20260925 12:15:15.685647 16321 catalog_manager.cc:4584] TS be01383017b64a0e9ffbd37159671917 (127.0.98.1:36863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88b15574c3304b4eba5165bd76104890: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:15:15.686242 16826 tablet_replica.cc:330] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: stopping tablet replica
I20260925 12:15:15.686364 16826 raft_consensus.cc:2229] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:15.686486 16826 raft_consensus.cc:2258] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:15.686988 16826 ts_tablet_manager.cc:1905] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:15.688457 16826 ts_tablet_manager.cc:1918] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:15.688609 16826 log.cc:1196] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/88b15574c3304b4eba5165bd76104890
I20260925 12:15:15.688891 16826 ts_tablet_manager.cc:1939] T 88b15574c3304b4eba5165bd76104890 P d5f020b3cfbc460f8e7debe76101110f: Deleting consensus metadata
I20260925 12:15:15.689385 16321 catalog_manager.cc:4841] TS d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): tablet 88b15574c3304b4eba5165bd76104890 (table test-table [id=e106951648fc40058ec3d34e9be95cba]) successfully deleted
I20260925 12:15:15.691877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16666
I20260925 12:15:15.699777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16304
I20260925 12:15:35.705754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--webserver_interface=127.0.98.62
--webserver_port=42431
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:35.794822 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:35.795114 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:35.795159 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:35.796676 16828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 12:15:35.796756 16828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 12:15:35.796780 16828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 12:15:35.796814 16828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:35.796850 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:35.796885 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:35.796919 16828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:35.796950 16828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:35.798723 16828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:35.799050 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:35.799306 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:15:35.804935 16828 server_base.cc:1034] running on GCE node
W20260925 12:15:35.804939 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:35.804957 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:35.804957 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:35.805586 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:35.805975 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:35.807166 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790338535807139 us; error 42 us; skew 500 ppm
Sep 25 12:15:35 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:35.819357 16828 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:15:35.819937 16828 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260925 12:15:35.820209 16828 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:35.820291 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:35.821784 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:35.822602 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:35.822826 16828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:35.822925 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ccf5fd333c0b49f1b34974882c96204f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "dad1f6a77dc808721c3f2733091bf3f6"
server_key_iv: "3b604745c8ae4ef7a5bf9b8e608e9a35"
server_key_version: "encryptionkey@0"
I20260925 12:15:35.823276 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:35.849223 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:35.851622 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:35.851738 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:35.855667 16828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43811
I20260925 12:15:35.855743 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43811 every 8 connection(s)
I20260925 12:15:35.856141 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:35.857697 16896 sys_catalog.cc:263] Verifying existing consensus state
I20260925 12:15:35.858398 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap starting.
I20260925 12:15:35.861817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16828
I20260925 12:15:35.862118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36863
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=38397
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:15:35.867859 16896 log.cc:824] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:35.870509 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:15:35.870864 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: Bootstrap complete.
I20260925 12:15:35.873274 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:35.873545 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Initialized, Role: FOLLOWER
I20260925 12:15:35.873770 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:35.873876 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:35.873932 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:35.873980 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 2 FOLLOWER]: Advancing to term 3
I20260925 12:15:35.874986 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:35.875119 16896 leader_election.cc:302] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf5fd333c0b49f1b34974882c96204f; no voters:
I20260925 12:15:35.875351 16896 leader_election.cc:288] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 12:15:35.875391 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 3 FOLLOWER]: Leader election won for term 3
I20260925 12:15:35.875536 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [term 3 LEADER]: Becoming Leader. State: Replica: ccf5fd333c0b49f1b34974882c96204f, State: Running, Role: LEADER
I20260925 12:15:35.875630 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:35.875684 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } }
I20260925 12:15:35.876031 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf5fd333c0b49f1b34974882c96204f. Latest consensus state: current_term: 3 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:15:35.876116 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:35.876667 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ccf5fd333c0b49f1b34974882c96204f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf5fd333c0b49f1b34974882c96204f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43811 } } }
I20260925 12:15:35.876741 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:35.877621 16917 catalog_manager.cc:1259] Loaded cluster ID: 30115476a8f74610bf251db35d828cb7
I20260925 12:15:35.877668 16917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: loading cluster ID for follower catalog manager: success
I20260925 12:15:35.879004 16917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccf5fd333c0b49f1b34974882c96204f: acquiring CA information for follower catalog manager: success
F20260925 12:15:35.879424 16917 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee5acd99ad google::LogMessage::Fail() at ??:0
@ 0x7fee5acdc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fee5acd94de google::LogMessage::Flush() at ??:0
@ 0x7fee5acdaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee5d446104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fee62c326b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fee62c32989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fee62c32e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fee62c358c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fee5cb9c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee5da006db start_thread at ??:0
@ 0x7fee597d761f clone at ??:0
W20260925 12:15:35.968910 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:35.969120 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:35.969152 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:35.970350 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:35.970404 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:35.970433 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:35.971951 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38397
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:35.972218 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:35.972481 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:35.977993 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:35.977993 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:35.977967 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:35.978403 16898 server_base.cc:1034] running on GCE node
I20260925 12:15:35.978623 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:35.978935 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:35.980109 16898 hybrid_clock.cc:648] HybridClock initialized: now 1790338535980090 us; error 38 us; skew 500 ppm
Sep 25 12:15:35 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:35.982869 16898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:15:35.983395 16898 webserver.cc:460] Webserver started at http://127.0.98.1:38397/ using document root <none> and password file <none>
I20260925 12:15:35.983631 16898 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:35.983707 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:35.985272 16898 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 12:15:35.986327 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:35.986572 16898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:35.986685 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "be01383017b64a0e9ffbd37159671917"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "34fa8e254ac2f1b5694e2b1e3bef8dc0"
server_key_iv: "86c47ded8d8891810b8c8144efbc5257"
server_key_version: "encryptionkey@0"
I20260925 12:15:35.987092 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:36.006146 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:36.008710 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:36.008878 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:36.009137 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:36.009613 16952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 12:15:36.011507 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 12:15:36.011576 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 12:15:36.011620 16898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 12:15:36.012283 16898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 12:15:36.012343 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 12:15:36.012421 16952 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Bootstrap starting.
I20260925 12:15:36.014349 16952 log.cc:824] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:36.017868 16952 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 12:15:36.018263 16952 tablet_bootstrap.cc:492] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Bootstrap complete.
I20260925 12:15:36.018497 16952 ts_tablet_manager.cc:1397] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260925 12:15:36.019537 16898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36863
I20260925 12:15:36.019694 17060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36863 every 8 connection(s)
I20260925 12:15:36.019902 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:36.021117 16952 raft_consensus.cc:348] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:36.021399 16952 raft_consensus.cc:729] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01383017b64a0e9ffbd37159671917, State: Initialized, Role: FOLLOWER
I20260925 12:15:36.021549 16952 consensus_queue.cc:260] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:36.021799 16952 ts_tablet_manager.cc:1428] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:15:36.028232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 16898
I20260925 12:15:36.028606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38815
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=38595
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:36.074651 17063 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:15:36.020700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:41575)
0925 12:15:36.020868 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.020920 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:36.021089 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:36.074443 (+ 53354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
W20260925 12:15:36.074931 17061 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43811 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43811: Client connection negotiation failed: client connection to 127.0.98.62:43811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 12:15:36.075320 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.075193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:48861)
0925 12:15:36.075256 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.075299 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 12:15:36.075814 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.075723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:39323)
0925 12:15:36.075772 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.075796 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 12:15:36.075970 17061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 12:15:36.114257 17065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:36.114462 17065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:36.114487 17065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:36.115675 17065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:36.115718 17065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:36.115746 17065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:36.117295 17065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38595
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:36.117542 17065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:36.117790 17065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:36.123065 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:36.123067 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:36.123283 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:36.123623 17065 server_base.cc:1034] running on GCE node
I20260925 12:15:36.123837 17065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:36.124115 17065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:36.125265 17065 hybrid_clock.cc:648] HybridClock initialized: now 1790338536125248 us; error 31 us; skew 500 ppm
Sep 25 12:15:36 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:36.127976 17065 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:15:36.128475 17065 webserver.cc:460] Webserver started at http://127.0.98.2:38595/ using document root <none> and password file <none>
I20260925 12:15:36.128758 17065 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:36.128829 17065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:36.130193 17065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:36.130959 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.131188 17065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:36.131295 17065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "b8d3292fbde44144abf28f7f3e7c76a9"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "09f70ab4a20af37e8e6422b444ff16c8"
server_key_iv: "846f57c898679fd70aeb5015417f5258"
server_key_version: "encryptionkey@0"
I20260925 12:15:36.131676 17065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:36.139998 17065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:36.142427 17065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:36.142603 17065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:36.142853 17065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:36.143165 17065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:36.143221 17065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.143261 17065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:36.143288 17065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.149432 17065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38815
I20260925 12:15:36.149474 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38815 every 8 connection(s)
I20260925 12:15:36.149823 17065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:15:36.150995 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.150546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:47123)
0925 12:15:36.150755 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.150849 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
W20260925 12:15:36.151525 17194 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43811 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43811: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
I20260925 12:15:36.151847 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.151727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:54517)
0925 12:15:36.151774 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.151819 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 12:15:36.152159 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.152077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:39895)
0925 12:15:36.152123 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.152145 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:15:36.152293 17194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:15:36.154068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17065
I20260925 12:15:36.154366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45005
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43811
--webserver_port=46717
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:36.238547 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:36.238772 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:36.238798 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:36.240041 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:36.240094 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:36.240123 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:36.241662 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46717
--tserver_master_addrs=127.0.98.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:36.241940 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:36.242210 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:36.247747 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:36.247782 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:36.247752 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:36.248095 17197 server_base.cc:1034] running on GCE node
I20260925 12:15:36.248345 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:36.248631 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:36.249783 17197 hybrid_clock.cc:648] HybridClock initialized: now 1790338536249772 us; error 29 us; skew 500 ppm
Sep 25 12:15:36 dist-test-slave-2hqq krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:36.252346 17197 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:15:36.252861 17197 webserver.cc:460] Webserver started at http://127.0.98.3:46717/ using document root <none> and password file <none>
I20260925 12:15:36.253090 17197 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:36.253162 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:36.254534 17197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:36.255187 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.255384 17197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:36.255505 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "d5f020b3cfbc460f8e7debe76101110f"
format_stamp: "Formatted at 2026-09-25 12:14:54 on dist-test-slave-2hqq"
server_key: "8fceaec4f10fa279454e0852d03643ce"
server_key_iv: "0dc0c01f15f635287f1897c21e9b05c3"
server_key_version: "encryptionkey@0"
I20260925 12:15:36.255890 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:36.274290 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:36.276925 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:36.277107 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:36.277366 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:36.277691 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:36.277750 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.277791 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:36.277812 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:36.283713 17197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260925 12:15:36.283764 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260925 12:15:36.284122 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:36.285305 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.284922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:34213)
0925 12:15:36.285086 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.285159 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
W20260925 12:15:36.285765 17326 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43811 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43811: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
I20260925 12:15:36.286104 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.285991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:46393)
0925 12:15:36.286045 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.286076 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:15:36.286448 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:36.286352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:56849)
0925 12:15:36.286408 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:36.286432 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 12:15:36.286580 17326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 12:15:36.289784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17197
I20260925 12:15:37.076818 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.076455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:55811)
0925 12:15:37.076691 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:37.076746 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":140,"threads_started":1}
I20260925 12:15:37.153019 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.152793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:40601)
0925 12:15:37.152924 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:37.152966 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 12:15:37.287308 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.287021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:53823)
0925 12:15:37.287159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:37.287201 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260925 12:15:37.859514 17333 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:37.859655 17333 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:37.860088 17333 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
Sep 25 12:15:37 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 12:15:37 dist-test-slave-2hqq krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 12:15:37.871975 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.860807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53539 (local address 127.0.98.2:38815)
0925 12:15:37.860948 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:37.860959 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:37.860987 (+ 28us) server_negotiation.cc:407] Connection header received
0925 12:15:37.861380 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:37.861382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:37.861418 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:37.861594 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:37.862526 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.863434 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.864408 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.864901 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.865040 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:37.867672 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:37.867678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:37.867678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:37.867690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:37.869806 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870431 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:37.870435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:37.870435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:37.870561 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870939 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:37.870940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:37.870940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:37.871259 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:37.871340 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:37.871715 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 12:15:37.871858 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 12:15:37.871975 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.860605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53213 (local address 127.0.98.3:45005)
0925 12:15:37.860814 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:37.860828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:37.860857 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:15:37.861337 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:37.861339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:37.861392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:37.861571 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:37.862367 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.863338 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.864408 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.864918 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.865064 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:37.868083 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:37.868087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:37.868087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:37.868097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:37.870142 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870586 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:37.870588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:37.870589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:37.870695 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870935 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:37.870936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:37.870936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:37.871254 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:37.871340 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:37.871715 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 12:15:37.871858 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260925 12:15:37.872042 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.860648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38815 (local address 127.0.98.1:53539)
0925 12:15:37.860749 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:37.860760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:37.860806 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:37.861605 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:37.861608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:37.861994 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:37.862376 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:37.862378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.863548 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.863549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:37.864307 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:37.864310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.865014 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.865015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:37.865146 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:37.865816 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:37.865820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:37.867521 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:37.869917 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:37.869923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:37.869925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:37.870163 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:37.870661 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:37.870662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:37.870853 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:37.871323 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:37.871331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:37.871506 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 12:15:37.871643 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 12:15:37.871896 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":101,"thread_start_us":53,"threads_started":1}
I20260925 12:15:37.872761 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:37.860414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.98.1:53213)
0925 12:15:37.860543 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:37.860556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:37.860628 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:37.861604 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:37.861607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:37.862036 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:37.862227 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:37.862234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.863472 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.863473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:37.864301 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:37.864304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:37.865029 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:37.865030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:37.865150 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:37.865762 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:37.865770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:37.867938 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:37.870284 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:37.870285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:37.870506 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:37.870783 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:37.870784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:37.870784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:37.870863 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:37.871329 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:37.871332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:37.871507 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 12:15:37.871643 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 12:15:37.871958 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":124,"thread_start_us":102,"threads_started":1}
W20260925 12:15:37.872994 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
W20260925 12:15:37.873658 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
I20260925 12:15:37.873725 16946 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be01383017b64a0e9ffbd37159671917; no voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f
I20260925 12:15:37.873832 17333 raft_consensus.cc:2726] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:15:38.077632 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:38.077476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:35859)
0925 12:15:38.077541 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:38.077588 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 12:15:38.153827 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:38.153599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:58131)
0925 12:15:38.153746 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:38.153785 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260925 12:15:38.288081 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:38.287820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:50691)
0925 12:15:38.287974 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:38.288029 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260925 12:15:39.078496 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:39.078264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:44359)
0925 12:15:39.078404 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:39.078447 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260925 12:15:39.154563 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:39.154337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:51319)
0925 12:15:39.154475 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:39.154518 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260925 12:15:39.288898 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:39.288670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:38123)
0925 12:15:39.288812 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:39.288852 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260925 12:15:39.862205 17343 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:39.862306 17343 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:39.862531 17343 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
W20260925 12:15:39.863168 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
W20260925 12:15:39.863248 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
I20260925 12:15:39.863266 16946 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be01383017b64a0e9ffbd37159671917; no voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f
I20260925 12:15:39.863345 17343 raft_consensus.cc:2726] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:15:40.079320 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:40.079098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:59837)
0925 12:15:40.079232 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:40.079273 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260925 12:15:40.155393 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:40.155165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:38843)
0925 12:15:40.155299 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:40.155337 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260925 12:15:40.289641 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:40.289374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:58497)
0925 12:15:40.289534 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:40.289578 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260925 12:15:41.080407 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:41.080145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:45889)
0925 12:15:41.080310 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:41.080355 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260925 12:15:41.156361 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:41.156086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:42753)
0925 12:15:41.156261 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:41.156312 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260925 12:15:41.290341 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:41.290095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:53535)
0925 12:15:41.290252 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:41.290295 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260925 12:15:42.081173 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:42.080954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:45657)
0925 12:15:42.081081 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:42.081124 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260925 12:15:42.157150 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:42.156922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:40419)
0925 12:15:42.157062 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:42.157104 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260925 12:15:42.291029 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:42.290809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:55821)
0925 12:15:42.290934 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:42.290982 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 12:15:42.611248 17353 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:42.611348 17353 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:42.611553 17353 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
W20260925 12:15:42.611970 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
W20260925 12:15:42.612071 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
I20260925 12:15:42.612108 16946 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be01383017b64a0e9ffbd37159671917; no voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f
I20260925 12:15:42.612207 17353 raft_consensus.cc:2726] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:15:43.081914 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:43.081705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:55793)
0925 12:15:43.081829 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:43.081869 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260925 12:15:43.157914 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:43.157660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:56177)
0925 12:15:43.157829 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:43.157869 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":125,"threads_started":1}
I20260925 12:15:43.291708 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:43.291482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:33445)
0925 12:15:43.291618 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:43.291662 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260925 12:15:44.082698 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:44.082430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:39059)
0925 12:15:44.082596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:44.082645 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260925 12:15:44.158599 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:44.158346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:46899)
0925 12:15:44.158512 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:44.158553 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260925 12:15:44.292409 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:44.292174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:58055)
0925 12:15:44.292326 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:44.292360 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260925 12:15:45.083554 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:45.083305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:39097)
0925 12:15:45.083455 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:45.083504 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260925 12:15:45.159322 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:45.159099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:33289)
0925 12:15:45.159236 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:45.159277 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260925 12:15:45.293129 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:45.292910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:56029)
0925 12:15:45.293043 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:45.293082 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260925 12:15:45.452186 17363 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:45.452287 17363 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:45.452570 17363 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
W20260925 12:15:45.453176 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
W20260925 12:15:45.453249 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
I20260925 12:15:45.453267 16946 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be01383017b64a0e9ffbd37159671917; no voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f
I20260925 12:15:45.453389 17363 raft_consensus.cc:2726] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:15:46.084436 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:46.084158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:35971)
0925 12:15:46.084323 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:46.084384 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260925 12:15:46.160092 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:46.159858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:60567)
0925 12:15:46.159995 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:46.160036 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260925 12:15:46.293834 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:46.293613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:49293)
0925 12:15:46.293753 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:46.293788 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260925 12:15:47.085269 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:47.085036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:47617)
0925 12:15:47.085175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:47.085220 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 12:15:47.160895 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:47.160651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:56893)
0925 12:15:47.160803 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:47.160848 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 12:15:47.294595 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:47.294382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:43619)
0925 12:15:47.294507 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:47.294547 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 12:15:48.086102 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:48.085883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:33435)
0925 12:15:48.086019 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:48.086054 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 12:15:48.161787 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:48.161548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:40381)
0925 12:15:48.161699 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:48.161741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260925 12:15:48.295442 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:48.295220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:56017)
0925 12:15:48.295350 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:48.295394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260925 12:15:49.086899 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:49.086616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:53179)
0925 12:15:49.086799 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:49.086849 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260925 12:15:49.162561 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:49.162299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:34051)
0925 12:15:49.162462 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:49.162514 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260925 12:15:49.296211 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:49.295963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:44163)
0925 12:15:49.296120 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:49.296164 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260925 12:15:50.087769 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:50.087523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:50711)
0925 12:15:50.087682 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:50.087722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260925 12:15:50.163329 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:50.163061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:42101)
0925 12:15:50.163243 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:50.163283 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260925 12:15:50.297055 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:50.296810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:48369)
0925 12:15:50.296949 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:50.297003 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260925 12:15:50.760003 17379 raft_consensus.cc:482] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:50.760128 17379 raft_consensus.cc:504] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f020b3cfbc460f8e7debe76101110f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "b8d3292fbde44144abf28f7f3e7c76a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38815 } } peers { permanent_uuid: "be01383017b64a0e9ffbd37159671917" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36863 } }
I20260925 12:15:50.760314 17379 leader_election.cc:288] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005), b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815)
W20260925 12:15:50.760916 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5f020b3cfbc460f8e7debe76101110f (127.0.98.3:45005): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
W20260925 12:15:50.761010 16946 leader_election.cc:341] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8d3292fbde44144abf28f7f3e7c76a9 (127.0.98.2:38815): Not found: Tablet not found: 88b15574c3304b4eba5165bd76104890
I20260925 12:15:50.761065 16946 leader_election.cc:302] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be01383017b64a0e9ffbd37159671917; no voters: b8d3292fbde44144abf28f7f3e7c76a9, d5f020b3cfbc460f8e7debe76101110f
I20260925 12:15:50.761173 17379 raft_consensus.cc:2726] T 88b15574c3304b4eba5165bd76104890 P be01383017b64a0e9ffbd37159671917 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 12:15:51.088634 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:51.088363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.1:46459)
0925 12:15:51.088538 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:51.088578 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260925 12:15:51.164204 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:51.163939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.2:36031)
0925 12:15:51.164103 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:51.164150 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 12:15:51.296133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 16898
I20260925 12:15:51.297947 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:51.297743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43811 (local address 127.0.98.3:34357)
0925 12:15:51.297867 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:51.297907 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260925 12:15:51.301162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17065
I20260925 12:15:51.305712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17197
2026-09-25T12:15:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58482 ms)
[----------] 30 tests from SecurityITest (260315 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:15:51 dist-test-slave-2hqq krb5kdc[17385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:15:51 dist-test-slave-2hqq krb5kdc[17385](info): set up 2 sockets
Sep 25 12:15:51 dist-test-slave-2hqq krb5kdc[17385](info): commencing operation
krb5kdc: starting...
W20260925 12:15:52.360265 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:15:52.483635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:52.565734 17398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:52.565953 17398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:52.565979 17398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:52.567071 17398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:52.567104 17398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:52.567118 17398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:52.567131 17398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:52.567143 17398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:52.568572 17398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:52.568759 17398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:52.568934 17398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:52.573871 17406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.573949 17403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.573920 17404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:52.573997 17398 server_base.cc:1034] running on GCE node
I20260925 12:15:52.574278 17398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:52.574533 17398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:52.575673 17398 hybrid_clock.cc:648] HybridClock initialized: now 1790338552575662 us; error 27 us; skew 500 ppm
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:52.578308 17398 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:15:52.578752 17398 webserver.cc:460] Webserver started at http://127.0.98.62:36493/ using document root <none> and password file <none>
I20260925 12:15:52.578972 17398 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:52.579034 17398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:52.579178 17398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:52.580255 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "7cfafc85587b4dc2840e1d67f8dcd873"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "a37704a88f5e3569ddee29eb7d1c6d43"
server_key_iv: "7ffba27620de3580c0460a44892d1b74"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.580649 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cfafc85587b4dc2840e1d67f8dcd873"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "a37704a88f5e3569ddee29eb7d1c6d43"
server_key_iv: "7ffba27620de3580c0460a44892d1b74"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.581979 17398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:52.582726 17413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.582895 17398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:52.582995 17398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "7cfafc85587b4dc2840e1d67f8dcd873"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "a37704a88f5e3569ddee29eb7d1c6d43"
server_key_iv: "7ffba27620de3580c0460a44892d1b74"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.583076 17398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:52.595119 17398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:52.597826 17398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:52.597998 17398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:52.602793 17398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41345
I20260925 12:15:52.602855 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41345 every 8 connection(s)
I20260925 12:15:52.603308 17398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:52.604413 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:52.607990 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: Bootstrap starting.
I20260925 12:15:52.608785 17466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:52.609119 17466 log.cc:824] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:52.609931 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: No bootstrap required, opened a new log
I20260925 12:15:52.611117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17398
I20260925 12:15:52.611241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:52.611488 392 external_mini_cluster.cc:1428] Setting key 895d2e82a5741f43f7c403c157364769
I20260925 12:15:52.612773 17466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } }
I20260925 12:15:52.613008 17466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:52.613088 17466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfafc85587b4dc2840e1d67f8dcd873, State: Initialized, Role: FOLLOWER
I20260925 12:15:52.613283 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } }
I20260925 12:15:52.613406 17466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:52.613464 17466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:52.613543 17466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:52.614367 17466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } }
I20260925 12:15:52.614490 17466 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfafc85587b4dc2840e1d67f8dcd873; no voters:
I20260925 12:15:52.614708 17466 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:52.614749 17471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:52.614941 17471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfafc85587b4dc2840e1d67f8dcd873, State: Running, Role: LEADER
I20260925 12:15:52.614996 17466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:52.615097 17471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } }
I20260925 12:15:52.615553 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfafc85587b4dc2840e1d67f8dcd873. Latest consensus state: current_term: 1 leader_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } } }
I20260925 12:15:52.615592 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfafc85587b4dc2840e1d67f8dcd873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41345 } } }
I20260925 12:15:52.615711 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:52.615895 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:52.615903 17480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:52.616475 17480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:52.618271 17480 catalog_manager.cc:1347] Generated new cluster ID: 386797a6ceff4a4ca367f9b8f23cff09
I20260925 12:15:52.618319 17480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:52.624019 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:52.612000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.62:41345)
0925 12:15:52.612180 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:52.612189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:52.612214 (+ 25us) server_negotiation.cc:407] Connection header received
0925 12:15:52.612726 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:52.612733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:52.612784 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:52.612959 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:52.613656 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.615023 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.616146 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.616817 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.616943 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:52.619843 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:52.619847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:52.619848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:52.619859 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:52.621975 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:52.622379 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:52.622381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:52.622382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:52.622526 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:52.622747 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:52.622748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:52.622748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:52.623050 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:52.623593 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:52.623672 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 12:15:52.623807 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260925 12:15:52.648365 17480 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:52.649197 17480 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:52.660281 17480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: Generated new TSK 0
I20260925 12:15:52.660514 17480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:15:52.696376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:52.792820 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:52.793061 17492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:52.793098 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:52.794265 17492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:52.794332 17492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:52.794378 17492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:52.795876 17492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:52.796142 17492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:52.796386 17492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:15:52.801882 17492 server_base.cc:1034] running on GCE node
W20260925 12:15:52.801849 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.802038 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.801856 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:52.802301 17492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:52.802490 17492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:52.803632 17492 hybrid_clock.cc:648] HybridClock initialized: now 1790338552803620 us; error 27 us; skew 500 ppm
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:52.806200 17492 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:15:52.806726 17492 webserver.cc:460] Webserver started at http://127.0.98.1:40727/ using document root <none> and password file <none>
I20260925 12:15:52.806972 17492 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:52.807015 17492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:52.807171 17492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:52.808233 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "c062324672e74d0a9636aa9b26991288"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "604b690c63afe5826f2ca73e59032b7b"
server_key_iv: "93e8f8cc655371a88d0ec93b2bdb3258"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.808611 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c062324672e74d0a9636aa9b26991288"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "604b690c63afe5826f2ca73e59032b7b"
server_key_iv: "93e8f8cc655371a88d0ec93b2bdb3258"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.809923 17492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:52.810644 17507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.810837 17492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:52.810940 17492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "c062324672e74d0a9636aa9b26991288"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "604b690c63afe5826f2ca73e59032b7b"
server_key_iv: "93e8f8cc655371a88d0ec93b2bdb3258"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.811010 17492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:52.839140 17492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:52.841632 17492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:52.841786 17492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:52.842067 17492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:52.842391 17492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:52.842427 17492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.842450 17492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:52.842463 17492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.848230 17492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35659
I20260925 12:15:52.848294 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35659 every 8 connection(s)
I20260925 12:15:52.848640 17492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:52.852276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17492
I20260925 12:15:52.852358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:52.852648 392 external_mini_cluster.cc:1428] Setting key 4a6143264985cfa845068d1473290151
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:52.859565 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:52.849422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34851 (local address 127.0.98.62:41345)
0925 12:15:52.849560 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:52.849563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:52.849763 (+ 200us) server_negotiation.cc:407] Connection header received
0925 12:15:52.850784 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:52.850786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:52.850798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:52.850850 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:52.851502 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.852014 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.853557 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.853772 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.853861 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:52.856302 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:52.856304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:52.856304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:52.856315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:52.858178 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:52.858577 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:52.858579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:52.858579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:52.858645 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:52.858890 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:52.858891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:52.858892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:52.859047 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:52.859122 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:52.859369 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 12:15:52.859463 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 12:15:52.859611 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:52.849378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41345 (local address 127.0.98.1:34851)
0925 12:15:52.849609 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:52.849662 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:52.850651 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:52.850923 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:52.850928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:52.851200 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:52.851389 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:52.851393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.852149 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.852150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:52.853453 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:52.853456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:52.853873 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:52.853874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:52.854020 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:52.854591 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:52.854602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:52.856210 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:52.858271 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:52.858278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:52.858280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:52.858496 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:52.858720 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:52.858721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:52.858721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:52.858821 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:52.859115 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:52.859120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:52.859238 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 12:15:52.859319 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:15:52.859438 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":125,"threads_started":1}
I20260925 12:15:52.860188 17621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41345
I20260925 12:15:52.860296 17621 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:52.860517 17621 heartbeater.cc:511] Master 127.0.98.62:41345 requested a full tablet report, sending...
I20260925 12:15:52.861088 17430 ts_manager.cc:194] Registered new tserver with Master: c062324672e74d0a9636aa9b26991288 (127.0.98.1:35659)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:15:52.861764 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:15:52.880131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:52.961020 17626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:52.961272 17626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:52.961309 17626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:52.962445 17626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:52.962510 17626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:52.962567 17626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:52.964041 17626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:52.964293 17626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:52.964581 17626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:52.969650 17631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.969601 17632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:52.969607 17634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:52.969974 17626 server_base.cc:1034] running on GCE node
I20260925 12:15:52.970242 17626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:52.970499 17626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:52.971657 17626 hybrid_clock.cc:648] HybridClock initialized: now 1790338552971647 us; error 32 us; skew 500 ppm
Sep 25 12:15:52 dist-test-slave-2hqq krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:52.974613 17626 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:15:52.975054 17626 webserver.cc:460] Webserver started at http://127.0.98.2:35629/ using document root <none> and password file <none>
I20260925 12:15:52.975291 17626 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:52.975361 17626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:52.975522 17626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:52.976703 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8be9a2f769d4079b73fe7f55791b6f4"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "6c753a885a4b7527d249f128d51a3cb5"
server_key_iv: "fcd07c84b6b32e9c5dd670f5627ada50"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.977072 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8be9a2f769d4079b73fe7f55791b6f4"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "6c753a885a4b7527d249f128d51a3cb5"
server_key_iv: "fcd07c84b6b32e9c5dd670f5627ada50"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.978439 17626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:15:52.979265 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.979552 17626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:52.979681 17626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "b8be9a2f769d4079b73fe7f55791b6f4"
format_stamp: "Formatted at 2026-09-25 12:15:52 on dist-test-slave-2hqq"
server_key: "6c753a885a4b7527d249f128d51a3cb5"
server_key_iv: "fcd07c84b6b32e9c5dd670f5627ada50"
server_key_version: "encryptionkey@0"
I20260925 12:15:52.979761 17626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:52.993909 17626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:52.996031 17626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:52.996206 17626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:52.996459 17626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:52.996868 17626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:52.996922 17626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:52.996958 17626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:52.996986 17626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.002635 17626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39069
I20260925 12:15:53.002727 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39069 every 8 connection(s)
I20260925 12:15:53.003047 17626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:15:53.004926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17626
I20260925 12:15:53.005038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:53.005276 392 external_mini_cluster.cc:1428] Setting key 465f10a270615f0df863db02ff30169f
Sep 25 12:15:53 dist-test-slave-2hqq krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:53.014447 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.003784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42975 (local address 127.0.98.62:41345)
0925 12:15:53.003856 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.003858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.004038 (+ 180us) server_negotiation.cc:407] Connection header received
0925 12:15:53.005159 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.005161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.005180 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.005268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.006112 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.006900 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.008220 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.008570 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.008647 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.011185 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.011187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.011187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:53.011196 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.012914 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.013336 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.013338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.013339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.013399 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.013708 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.013709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.013709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.013878 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.013962 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.014252 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.014348 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:15:53.014586 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.003751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41345 (local address 127.0.98.2:42975)
0925 12:15:53.003911 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.003954 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.004999 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.005324 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.005328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.005747 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:53.006023 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.006030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.007014 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.007015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.008132 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.008135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.008662 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.008663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.008768 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.009313 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:53.009322 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:53.011078 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:53.013007 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:53.013014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:53.013015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:53.013256 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:53.013469 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:53.013470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:53.013470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:53.013639 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:53.013953 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:53.013958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:53.014096 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.014193 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.014322 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 12:15:53.015074 17755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41345
I20260925 12:15:53.015187 17755 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:53.015364 17755 heartbeater.cc:511] Master 127.0.98.62:41345 requested a full tablet report, sending...
I20260925 12:15:53.015827 17430 ts_manager.cc:194] Registered new tserver with Master: b8be9a2f769d4079b73fe7f55791b6f4 (127.0.98.2:39069)
I20260925 12:15:53.016240 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:15:53.032433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:53.113677 17760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:53.113960 17760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:53.113991 17760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:53.115257 17760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:53.115295 17760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:53.115322 17760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:53.117117 17760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:53.117311 17760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:53.117492 17760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:53.122627 17768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.122764 17760 server_base.cc:1034] running on GCE node
W20260925 12:15:53.122654 17765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.122627 17766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.123169 17760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:53.123379 17760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:53.124573 17760 hybrid_clock.cc:648] HybridClock initialized: now 1790338553124558 us; error 29 us; skew 500 ppm
Sep 25 12:15:53 dist-test-slave-2hqq krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:53.127588 17760 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:15:53.128074 17760 webserver.cc:460] Webserver started at http://127.0.98.3:44255/ using document root <none> and password file <none>
I20260925 12:15:53.128304 17760 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:53.128346 17760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:53.128439 17760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:53.129575 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff1dd1d177014d47aecd6aab4fec637a"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "6eb5d34ef534c52133585fbf71e8e7c2"
server_key_iv: "70b3d4a3ace04458493a3266aff7221f"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.129900 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff1dd1d177014d47aecd6aab4fec637a"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "6eb5d34ef534c52133585fbf71e8e7c2"
server_key_iv: "70b3d4a3ace04458493a3266aff7221f"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.131184 17760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:15:53.131850 17775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.132020 17760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:53.132149 17760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "ff1dd1d177014d47aecd6aab4fec637a"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "6eb5d34ef534c52133585fbf71e8e7c2"
server_key_iv: "70b3d4a3ace04458493a3266aff7221f"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.132228 17760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:53.155525 17760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:53.158098 17760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:53.158277 17760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:53.158543 17760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:53.158864 17760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:53.158919 17760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.158954 17760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:53.158996 17760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.164956 17760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41647
I20260925 12:15:53.165010 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41647 every 8 connection(s)
I20260925 12:15:53.165357 17760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:53.167735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17760
I20260925 12:15:53.167840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:53.168032 392 external_mini_cluster.cc:1428] Setting key 449ff964df1eef0b197275955bc2cde8
Sep 25 12:15:53 dist-test-slave-2hqq krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:53.174888 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.166160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46715 (local address 127.0.98.62:41345)
0925 12:15:53.166218 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.166220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.166385 (+ 165us) server_negotiation.cc:407] Connection header received
0925 12:15:53.167299 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.167300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.167309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.167367 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.168205 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.168920 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.169944 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.170131 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.170193 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.172625 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.172627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.172627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:53.172634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.173781 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.174133 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.174133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.174134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.174182 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.174401 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.174402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.174402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.174499 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.174548 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.174768 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.174825 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:15:53.175127 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.166121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41345 (local address 127.0.98.3:46715)
0925 12:15:53.166276 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.166316 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.167205 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.167420 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.167423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.167777 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:53.168082 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.168088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.169005 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.169006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.169867 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.169870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.170205 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.170205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.170301 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.170790 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:53.170797 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:53.172539 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:53.173850 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:53.173855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:53.173857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:53.174066 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:53.174254 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:53.174254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:53.174255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:53.174348 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:53.174550 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:53.174554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:53.174665 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.174750 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.174863 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 12:15:53.175613 17889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41345
I20260925 12:15:53.175727 17889 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:53.175874 17889 heartbeater.cc:511] Master 127.0.98.62:41345 requested a full tablet report, sending...
I20260925 12:15:53.176208 17430 ts_manager.cc:194] Registered new tserver with Master: ff1dd1d177014d47aecd6aab4fec637a (127.0.98.3:41647)
I20260925 12:15:53.176664 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46715
I20260925 12:15:53.179780 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:53.187638 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.181138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.62:41345)
0925 12:15:53.181199 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.181201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.181323 (+ 122us) server_negotiation.cc:407] Connection header received
0925 12:15:53.181398 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.181399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.181409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.181464 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.182030 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.182522 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.183273 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.183470 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.183551 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.184701 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.184702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.184703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:53.184710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.186239 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.186619 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.186621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.186621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.186680 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:53.186946 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:53.186947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:53.186947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:53.187177 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.187283 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.187463 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.187539 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 12:15:53.189359 17430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:53.190964 17430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:53.197510 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35659 (local address 127.0.0.1:34114)
0925 12:15:53.193541 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.193580 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.193651 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.194119 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.194119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.194121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.194160 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.194161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195486 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.195488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.196406 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.196427 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197202 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.197281 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197296 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.197364 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.197421 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":20,"thread_start_us":56,"threads_started":1}
I20260925 12:15:53.197572 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39069 (local address 127.0.0.1:52466)
0925 12:15:53.193641 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.193650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.193705 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.194074 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.194077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.194079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.194168 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.194170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195116 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.195118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.196369 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.196372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197328 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.197388 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197391 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.197440 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.197470 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":12,"thread_start_us":100,"threads_started":1}
I20260925 12:15:53.197736 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.0.98.2:39069)
0925 12:15:53.193853 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.193857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.193867 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:53.193905 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.193907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.193924 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.194249 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.194337 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195016 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.196457 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197219 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197382 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.197452 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.197494 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":125,"threads_started":1}
I20260925 12:15:53.197824 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41647 (local address 127.0.0.1:36496)
0925 12:15:53.193755 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.193763 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.193799 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.194079 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.194080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.194082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.194159 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.194161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195475 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.195476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.196629 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.196631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197533 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.197610 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197616 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.197673 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.197735 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":28,"thread_start_us":93,"threads_started":1}
I20260925 12:15:53.198099 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.98.3:41647)
0925 12:15:53.193807 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.193812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.193823 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:53.193870 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.193873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.193889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.193976 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.194259 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195378 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.196715 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197392 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197452 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197500 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.197975 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.198026 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260925 12:15:53.198107 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.193477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.98.1:35659)
0925 12:15:53.193897 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.193901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.193910 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:53.193949 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.193951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.193965 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.194035 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.194243 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.195361 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.196531 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.197098 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.197207 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.197270 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.197983 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.198026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":82,"threads_started":1}
I20260925 12:15:53.198511 17689 tablet_service.cc:1467] Processing CreateTablet for tablet f050083e6c33406bad68224fd27d1c1a (DEFAULT_TABLE table=test-table [id=f8a552a8e2fd42f785bfd933b9d5e20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.198650 17555 tablet_service.cc:1467] Processing CreateTablet for tablet f050083e6c33406bad68224fd27d1c1a (DEFAULT_TABLE table=test-table [id=f8a552a8e2fd42f785bfd933b9d5e20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.198750 17823 tablet_service.cc:1467] Processing CreateTablet for tablet f050083e6c33406bad68224fd27d1c1a (DEFAULT_TABLE table=test-table [id=f8a552a8e2fd42f785bfd933b9d5e20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.198871 17689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050083e6c33406bad68224fd27d1c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.198920 17555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050083e6c33406bad68224fd27d1c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.199081 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f050083e6c33406bad68224fd27d1c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.202162 17907 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: Bootstrap starting.
I20260925 12:15:53.202255 17906 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Bootstrap starting.
I20260925 12:15:53.202795 17906 tablet_bootstrap.cc:654] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.202800 17907 tablet_bootstrap.cc:654] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.203264 17907 log.cc:824] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.203294 17906 log.cc:824] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.203608 17908 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Bootstrap starting.
I20260925 12:15:53.203955 17907 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: No bootstrap required, opened a new log
I20260925 12:15:53.204010 17907 ts_tablet_manager.cc:1397] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:53.204021 17906 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: No bootstrap required, opened a new log
I20260925 12:15:53.204089 17906 ts_tablet_manager.cc:1397] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:53.205175 17908 tablet_bootstrap.cc:654] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.205713 17908 log.cc:824] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.205744 17906 raft_consensus.cc:348] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.205947 17906 raft_consensus.cc:374] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.206002 17906 raft_consensus.cc:729] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1dd1d177014d47aecd6aab4fec637a, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.206148 17906 consensus_queue.cc:260] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.206135 17907 raft_consensus.cc:348] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.206295 17907 raft_consensus.cc:374] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.206333 17907 raft_consensus.cc:729] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c062324672e74d0a9636aa9b26991288, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.206439 17906 ts_tablet_manager.cc:1428] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:15:53.206439 17907 consensus_queue.cc:260] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.206561 17908 tablet_bootstrap.cc:492] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: No bootstrap required, opened a new log
I20260925 12:15:53.206578 17889 heartbeater.cc:503] Master 127.0.98.62:41345 was elected leader, sending a full tablet report...
I20260925 12:15:53.206630 17908 ts_tablet_manager.cc:1397] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:15:53.206768 17621 heartbeater.cc:503] Master 127.0.98.62:41345 was elected leader, sending a full tablet report...
I20260925 12:15:53.206718 17907 ts_tablet_manager.cc:1428] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:53.208622 17908 raft_consensus.cc:348] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.208765 17908 raft_consensus.cc:374] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.208787 17908 raft_consensus.cc:729] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8be9a2f769d4079b73fe7f55791b6f4, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.208894 17908 consensus_queue.cc:260] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.209161 17908 ts_tablet_manager.cc:1428] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:15:53.209169 17755 heartbeater.cc:503] Master 127.0.98.62:41345 was elected leader, sending a full tablet report...
W20260925 12:15:53.253576 17756 tablet.cc:2369] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:53.311381 17912 raft_consensus.cc:482] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:53.311501 17912 raft_consensus.cc:504] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.311918 17912 leader_election.cc:288] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8be9a2f769d4079b73fe7f55791b6f4 (127.0.98.2:39069), c062324672e74d0a9636aa9b26991288 (127.0.98.1:35659)
I20260925 12:15:53.315009 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.312229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42521 (local address 127.0.98.1:35659)
0925 12:15:53.312281 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.312283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.312293 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:53.312323 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.312324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.312336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.312408 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.312780 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.313279 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314273 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.314635 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314740 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.314750 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.314903 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.314942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 12:15:53.315009 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.312117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35659 (local address 127.0.98.3:42521)
0925 12:15:53.312174 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.312186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.312228 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.312578 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.312578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.312580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.312633 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.312634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.313362 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.313363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.314156 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.314158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.314738 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.314812 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.314814 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.314894 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.314940 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 12:15:53.315068 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.312200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55041 (local address 127.0.98.2:39069)
0925 12:15:53.312247 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.312249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.312410 (+ 161us) server_negotiation.cc:407] Connection header received
0925 12:15:53.312449 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.312450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.312461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.312536 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.312683 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.313269 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314293 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.314744 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314807 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.314816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.314964 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.314999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:15:53.315316 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.312117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39069 (local address 127.0.98.3:55041)
0925 12:15:53.312330 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.312341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.312398 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.312544 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.312545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.312547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.312614 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.312616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.313398 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.313398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.314203 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.314206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.314808 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.314809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.314889 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.314892 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.314960 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.314998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":11,"thread_start_us":144,"threads_started":1}
I20260925 12:15:53.315336 17575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050083e6c33406bad68224fd27d1c1a" candidate_uuid: "ff1dd1d177014d47aecd6aab4fec637a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c062324672e74d0a9636aa9b26991288" is_pre_election: true
I20260925 12:15:53.315523 17575 raft_consensus.cc:2454] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1dd1d177014d47aecd6aab4fec637a in term 0.
I20260925 12:15:53.315785 17778 leader_election.cc:302] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c062324672e74d0a9636aa9b26991288, ff1dd1d177014d47aecd6aab4fec637a; no voters:
I20260925 12:15:53.315937 17912 raft_consensus.cc:2781] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:53.315907 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050083e6c33406bad68224fd27d1c1a" candidate_uuid: "ff1dd1d177014d47aecd6aab4fec637a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" is_pre_election: true
I20260925 12:15:53.315999 17912 raft_consensus.cc:482] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:53.316053 17912 raft_consensus.cc:3037] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:53.316056 17709 raft_consensus.cc:2454] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1dd1d177014d47aecd6aab4fec637a in term 0.
I20260925 12:15:53.316921 17912 raft_consensus.cc:504] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.317080 17912 leader_election.cc:288] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [CANDIDATE]: Term 1 election: Requested vote from peers b8be9a2f769d4079b73fe7f55791b6f4 (127.0.98.2:39069), c062324672e74d0a9636aa9b26991288 (127.0.98.1:35659)
I20260925 12:15:53.317217 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050083e6c33406bad68224fd27d1c1a" candidate_uuid: "ff1dd1d177014d47aecd6aab4fec637a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8be9a2f769d4079b73fe7f55791b6f4"
I20260925 12:15:53.317248 17575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f050083e6c33406bad68224fd27d1c1a" candidate_uuid: "ff1dd1d177014d47aecd6aab4fec637a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c062324672e74d0a9636aa9b26991288"
I20260925 12:15:53.317301 17709 raft_consensus.cc:3037] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:53.317312 17575 raft_consensus.cc:3037] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:53.318178 17709 raft_consensus.cc:2454] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1dd1d177014d47aecd6aab4fec637a in term 1.
I20260925 12:15:53.318186 17575 raft_consensus.cc:2454] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1dd1d177014d47aecd6aab4fec637a in term 1.
I20260925 12:15:53.318377 17778 leader_election.cc:302] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c062324672e74d0a9636aa9b26991288, ff1dd1d177014d47aecd6aab4fec637a; no voters:
I20260925 12:15:53.318461 17912 raft_consensus.cc:2781] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:53.318610 17912 raft_consensus.cc:686] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 1 LEADER]: Becoming Leader. State: Replica: ff1dd1d177014d47aecd6aab4fec637a, State: Running, Role: LEADER
I20260925 12:15:53.318717 17912 consensus_queue.cc:237] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } }
I20260925 12:15:53.319557 17429 catalog_manager.cc:5495] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1dd1d177014d47aecd6aab4fec637a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff1dd1d177014d47aecd6aab4fec637a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff1dd1d177014d47aecd6aab4fec637a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41647 } health_report { overall_health: HEALTHY } } }
I20260925 12:15:53.342542 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.340015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.98.3:41647)
0925 12:15:53.340081 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.340084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.340094 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:53.340190 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.340190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.340209 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.340292 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:53.340430 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.340957 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.341727 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.341924 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.342009 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.342177 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:53.342283 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:53.342381 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.342399 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.342417 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:15:53.344417 17709 raft_consensus.cc:1261] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 1 FOLLOWER]: Refusing update from remote peer ff1dd1d177014d47aecd6aab4fec637a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:53.344419 17575 raft_consensus.cc:1261] T f050083e6c33406bad68224fd27d1c1a P c062324672e74d0a9636aa9b26991288 [term 1 FOLLOWER]: Refusing update from remote peer ff1dd1d177014d47aecd6aab4fec637a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:53.344834 17916 consensus_queue.cc:1035] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8be9a2f769d4079b73fe7f55791b6f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:53.344973 17912 consensus_queue.cc:1035] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c062324672e74d0a9636aa9b26991288" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:53.347797 17920 mvcc.cc:205] Tried to move back new op lower bound from 7333226714495270912 to 7333226714393812992. Current Snapshot: MvccSnapshot[applied={T|T < 7333226714495270912}]
I20260925 12:15:53.349673 17430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:53.349771 17430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:53.350556 17430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cfafc85587b4dc2840e1d67f8dcd873: Sending DeleteTablet for 3 replicas of tablet f050083e6c33406bad68224fd27d1c1a
I20260925 12:15:53.350883 17689 tablet_service.cc:1514] Processing DeleteTablet for tablet f050083e6c33406bad68224fd27d1c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52466
I20260925 12:15:53.350984 17555 tablet_service.cc:1514] Processing DeleteTablet for tablet f050083e6c33406bad68224fd27d1c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260925 12:15:53.351091 17823 tablet_service.cc:1514] Processing DeleteTablet for tablet f050083e6c33406bad68224fd27d1c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36496
I20260925 12:15:53.351351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17492
I20260925 12:15:53.351480 17928 tablet_replica.cc:330] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: stopping tablet replica
I20260925 12:15:53.351568 17928 raft_consensus.cc:2229] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:53.351678 17928 raft_consensus.cc:2258] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:53.351399 17930 tablet_replica.cc:330] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: stopping tablet replica
I20260925 12:15:53.352048 17930 raft_consensus.cc:2229] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:15:53.352146 17930 raft_consensus.cc:2258] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:53.353489 17928 ts_tablet_manager.cc:1905] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:53.353684 17930 ts_tablet_manager.cc:1905] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:53.355648 17930 ts_tablet_manager.cc:1918] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:53.355726 17930 log.cc:1196] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/f050083e6c33406bad68224fd27d1c1a
I20260925 12:15:53.355835 17928 ts_tablet_manager.cc:1918] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:53.355957 17928 log.cc:1196] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/f050083e6c33406bad68224fd27d1c1a
I20260925 12:15:53.356014 17930 ts_tablet_manager.cc:1939] T f050083e6c33406bad68224fd27d1c1a P ff1dd1d177014d47aecd6aab4fec637a: Deleting consensus metadata
I20260925 12:15:53.356323 17928 ts_tablet_manager.cc:1939] T f050083e6c33406bad68224fd27d1c1a P b8be9a2f769d4079b73fe7f55791b6f4: Deleting consensus metadata
I20260925 12:15:53.356917 17414 catalog_manager.cc:4841] TS b8be9a2f769d4079b73fe7f55791b6f4 (127.0.98.2:39069): tablet f050083e6c33406bad68224fd27d1c1a (table test-table [id=f8a552a8e2fd42f785bfd933b9d5e20b]) successfully deleted
I20260925 12:15:53.356935 17417 catalog_manager.cc:4841] TS ff1dd1d177014d47aecd6aab4fec637a (127.0.98.3:41647): tablet f050083e6c33406bad68224fd27d1c1a (table test-table [id=f8a552a8e2fd42f785bfd933b9d5e20b]) successfully deleted
W20260925 12:15:53.358567 17416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35659 (error 108)
I20260925 12:15:53.358754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17626
I20260925 12:15:53.359630 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.359509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35659 (local address 127.0.0.1:34126)
0925 12:15:53.359557 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.359590 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:15:53.359760 17416 catalog_manager.cc:4584] TS c062324672e74d0a9636aa9b26991288 (127.0.98.1:35659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f050083e6c33406bad68224fd27d1c1a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35659: connect: Connection refused (error 111)
I20260925 12:15:53.364153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17760
I20260925 12:15:53.368930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17398
2026-09-25T12:15:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2063 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T12:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:53Z Disabled control of system clock
I20260925 12:15:53.396832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45769
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:53.477030 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:53.477254 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:53.477274 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:53.478395 17935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:53.478433 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:53.478446 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:53.478458 17935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:53.478470 17935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:53.479728 17935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:53.479933 17935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:53.480168 17935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:53.485054 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.485141 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.485261 17935 server_base.cc:1034] running on GCE node
W20260925 12:15:53.485154 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.485631 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:53.485975 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:53.487145 17935 hybrid_clock.cc:648] HybridClock initialized: now 1790338553487128 us; error 40 us; skew 500 ppm
I20260925 12:15:53.488651 17935 webserver.cc:460] Webserver started at http://127.0.98.62:42355/ using document root <none> and password file <none>
I20260925 12:15:53.488931 17935 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:53.489003 17935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:53.489156 17935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:53.490242 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "77828f972f754bd7b1106ea1f24038e0"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "4bccb490a5e2ded6fa1d34ae708afc66"
server_key_iv: "80e486c5e8c506e997dc7a4f846d64ff"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.490613 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "77828f972f754bd7b1106ea1f24038e0"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "4bccb490a5e2ded6fa1d34ae708afc66"
server_key_iv: "80e486c5e8c506e997dc7a4f846d64ff"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.491946 17935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:15:53.492686 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.492865 17935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:53.492965 17935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "77828f972f754bd7b1106ea1f24038e0"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "4bccb490a5e2ded6fa1d34ae708afc66"
server_key_iv: "80e486c5e8c506e997dc7a4f846d64ff"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.493048 17935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:53.504161 17935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:53.504454 17935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:53.504626 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:53.508906 17935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45769
I20260925 12:15:53.508931 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45769 every 8 connection(s)
I20260925 12:15:53.509281 17935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:53.510214 18002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.511152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 17935
I20260925 12:15:53.511256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:53.511507 392 external_mini_cluster.cc:1428] Setting key 61e69eba8fc8f4fcd0371e845aa0d64c
I20260925 12:15:53.513608 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: Bootstrap starting.
I20260925 12:15:53.514278 18002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.514591 18002 log.cc:824] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.515352 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: No bootstrap required, opened a new log
I20260925 12:15:53.517149 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.512055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.0.98.62:45769)
0925 12:15:53.512253 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.512260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.512285 (+ 25us) server_negotiation.cc:407] Connection header received
0925 12:15:53.512761 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.512767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.512828 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.512946 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.513165 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.514065 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.515141 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.515576 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.515663 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.516335 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.516338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.516338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:53.516551 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.516584 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:53.516595 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.516646 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.516756 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.516839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 12:15:53.517198 18002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } }
I20260925 12:15:53.517390 18002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.517431 18002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77828f972f754bd7b1106ea1f24038e0, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.517542 18002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } }
I20260925 12:15:53.517628 18002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:53.517681 18002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:53.517729 18002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:53.518302 18002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } }
I20260925 12:15:53.518419 18002 leader_election.cc:302] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77828f972f754bd7b1106ea1f24038e0; no voters:
I20260925 12:15:53.518580 18002 leader_election.cc:288] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:53.518615 18007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:53.518801 18007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [term 1 LEADER]: Becoming Leader. State: Replica: 77828f972f754bd7b1106ea1f24038e0, State: Running, Role: LEADER
I20260925 12:15:53.518839 18002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:53.518891 18007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } }
I20260925 12:15:53.519256 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77828f972f754bd7b1106ea1f24038e0. Latest consensus state: current_term: 1 leader_uuid: "77828f972f754bd7b1106ea1f24038e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } } }
I20260925 12:15:53.519266 18008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77828f972f754bd7b1106ea1f24038e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77828f972f754bd7b1106ea1f24038e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45769 } } }
I20260925 12:15:53.519385 18008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:53.519374 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:53.519661 18014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:53.520324 18014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:53.521905 18014 catalog_manager.cc:1347] Generated new cluster ID: 582204ce706b4d638cfd7bf629ec03ee
I20260925 12:15:53.521950 18014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:15:53.534484 18014 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:53.535194 18014 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:53.538275 18014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: Generated new TSK 0
I20260925 12:15:53.538462 18014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:15:53.548228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:53.628439 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:53.628703 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:53.628742 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:53.629904 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:53.629968 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:53.630012 18026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:53.631371 18026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:53.631628 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:53.631922 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:53.638523 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.638758 18026 server_base.cc:1034] running on GCE node
W20260925 12:15:53.638523 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.638649 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.639071 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:53.639359 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:53.640545 18026 hybrid_clock.cc:648] HybridClock initialized: now 1790338553640533 us; error 30 us; skew 500 ppm
I20260925 12:15:53.641956 18026 webserver.cc:460] Webserver started at http://127.0.98.1:38005/ using document root <none> and password file <none>
I20260925 12:15:53.642215 18026 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:53.642274 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:53.642411 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:53.643858 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b6661eb69ba4c0bb80fabb602bb7504"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "52bd1645cd33faaf94f4f90f90e56ecd"
server_key_iv: "082ef9ef27eae526cf86ee9570d6b7fc"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.644280 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b6661eb69ba4c0bb80fabb602bb7504"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "52bd1645cd33faaf94f4f90f90e56ecd"
server_key_iv: "082ef9ef27eae526cf86ee9570d6b7fc"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.645879 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 12:15:53.646713 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.646904 18026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:53.646992 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "9b6661eb69ba4c0bb80fabb602bb7504"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "52bd1645cd33faaf94f4f90f90e56ecd"
server_key_iv: "082ef9ef27eae526cf86ee9570d6b7fc"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.647058 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:53.672672 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:53.672969 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:53.673126 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:53.673350 18026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:53.673713 18026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:53.673748 18026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.673804 18026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:53.673830 18026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.679442 18026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260925 12:15:53.679522 18153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260925 12:15:53.679873 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:53.683383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18026
I20260925 12:15:53.683461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:53.683682 392 external_mini_cluster.cc:1428] Setting key 78973c6fe719d085beded325bacf44e7
I20260925 12:15:53.684755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:15:53.685598 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.680754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46643 (local address 127.0.98.62:45769)
0925 12:15:53.680822 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.680824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.681129 (+ 305us) server_negotiation.cc:407] Connection header received
0925 12:15:53.681622 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.681623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.681636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.681689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.682322 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.682821 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.683967 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.684237 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.684309 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.685301 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.685302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.685303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:53.685308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.685323 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:53.685327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.685382 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.685478 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.685508 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:15:53.685811 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.680745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45769 (local address 127.0.98.1:46643)
0925 12:15:53.680981 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.681017 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.681515 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.681751 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.681755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.681763 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:53.682253 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.682256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.682901 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.682902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.683883 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.683885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.684332 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.684333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.684525 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.685153 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:53.685154 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:53.685182 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:53.685203 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:53.685384 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:53.685389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:53.685393 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.685468 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.685573 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260925 12:15:53.686474 18154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45769
I20260925 12:15:53.686583 18154 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:53.686771 18154 heartbeater.cc:511] Master 127.0.98.62:45769 requested a full tablet report, sending...
I20260925 12:15:53.687279 17966 ts_manager.cc:194] Registered new tserver with Master: 9b6661eb69ba4c0bb80fabb602bb7504 (127.0.98.1:45115)
I20260925 12:15:53.687981 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46643
W20260925 12:15:53.767272 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:53.767560 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:53.767599 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:53.768803 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:53.768980 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:53.769039 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:53.770440 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:53.770713 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:53.771024 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:53.776142 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.776129 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.776188 18157 server_base.cc:1034] running on GCE node
W20260925 12:15:53.776129 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.776571 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:53.776777 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:53.777978 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790338553777965 us; error 27 us; skew 500 ppm
I20260925 12:15:53.779271 18157 webserver.cc:460] Webserver started at http://127.0.98.2:34507/ using document root <none> and password file <none>
I20260925 12:15:53.779513 18157 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:53.779558 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:53.779704 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:53.780910 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "653b2af2e5044a45a7e8901c6d055a00"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "8b6142c6f4724bc68bb620e2f3409a7b"
server_key_iv: "35f62a9422f3f306be78c0ba47493546"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.781268 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "653b2af2e5044a45a7e8901c6d055a00"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "8b6142c6f4724bc68bb620e2f3409a7b"
server_key_iv: "35f62a9422f3f306be78c0ba47493546"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.782557 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:53.783293 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.783478 18157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:53.783547 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "653b2af2e5044a45a7e8901c6d055a00"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "8b6142c6f4724bc68bb620e2f3409a7b"
server_key_iv: "35f62a9422f3f306be78c0ba47493546"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.783634 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:53.801690 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:53.802018 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:53.802173 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:53.802403 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:53.802733 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:53.802767 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.802788 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:53.802838 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.808480 18157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260925 12:15:53.808663 18284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260925 12:15:53.808959 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:15:53.809871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18157
I20260925 12:15:53.809971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:53.810250 392 external_mini_cluster.cc:1428] Setting key a14b68ecde5861eca19c0ac8d96ab051
I20260925 12:15:53.811515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 12:15:53.815049 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.810477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44667 (local address 127.0.98.62:45769)
0925 12:15:53.810652 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.810654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.810668 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:15:53.810775 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.810776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.810789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.810848 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.811753 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.812268 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.813489 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.813781 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.813838 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.814752 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.814753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.814754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:53.814757 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.814793 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:53.814797 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.814857 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.814964 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.814987 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 12:15:53.815349 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.809711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45769 (local address 127.0.98.2:44667)
0925 12:15:53.809908 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.809960 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.810632 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.810900 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.810903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.810911 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:53.811633 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.811642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.812369 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.812370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.813367 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.813370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.813876 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.813877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.814038 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.814597 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:53.814598 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:53.814635 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:53.814664 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:53.814873 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:53.814879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:53.814882 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.814963 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.815083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260925 12:15:53.815897 18285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45769
I20260925 12:15:53.815997 18285 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:53.816201 18285 heartbeater.cc:511] Master 127.0.98.62:45769 requested a full tablet report, sending...
I20260925 12:15:53.816602 17966 ts_manager.cc:194] Registered new tserver with Master: 653b2af2e5044a45a7e8901c6d055a00 (127.0.98.2:42729)
I20260925 12:15:53.817047 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44667
W20260925 12:15:53.896349 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:53.896616 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:53.896656 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:53.897877 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:53.897944 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:53.897989 18288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:53.899351 18288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:53.899616 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:53.899917 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:53.905429 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.905436 18294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:53.905439 18296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:53.905670 18288 server_base.cc:1034] running on GCE node
I20260925 12:15:53.905865 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:53.906066 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:53.907204 18288 hybrid_clock.cc:648] HybridClock initialized: now 1790338553907192 us; error 27 us; skew 500 ppm
I20260925 12:15:53.908450 18288 webserver.cc:460] Webserver started at http://127.0.98.3:33869/ using document root <none> and password file <none>
I20260925 12:15:53.908715 18288 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:53.908792 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:53.908946 18288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:53.910089 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e4ef3210904436387eea114b81be3ab"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "05e2fcf3ee741d66de68155d6636b8fd"
server_key_iv: "21af135ff42cb3177935d18de1e97ab5"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.910466 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4ef3210904436387eea114b81be3ab"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "05e2fcf3ee741d66de68155d6636b8fd"
server_key_iv: "21af135ff42cb3177935d18de1e97ab5"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.911787 18288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:15:53.912568 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.912763 18288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:53.912863 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "1e4ef3210904436387eea114b81be3ab"
format_stamp: "Formatted at 2026-09-25 12:15:53 on dist-test-slave-2hqq"
server_key: "05e2fcf3ee741d66de68155d6636b8fd"
server_key_iv: "21af135ff42cb3177935d18de1e97ab5"
server_key_version: "encryptionkey@0"
I20260925 12:15:53.912947 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:53.945792 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:53.946151 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:53.946316 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:53.946599 18288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:53.946976 18288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:53.947041 18288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.947086 18288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:53.947115 18288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:53.953022 18288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34823
I20260925 12:15:53.953080 18415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34823 every 8 connection(s)
I20260925 12:15:53.953449 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:53.960520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18288
I20260925 12:15:53.960635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:53.960848 392 external_mini_cluster.cc:1428] Setting key 2fc8d6d9c45e374cf4423f774c1c92d7
I20260925 12:15:53.966452 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.961141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50951 (local address 127.0.98.62:45769)
0925 12:15:53.962064 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.962080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.962088 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:53.962112 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.962113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.962124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.962170 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.963181 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.963861 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.964965 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.965197 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.965253 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.966161 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.966162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.966163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:53.966167 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.966180 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:53.966184 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.966231 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.966371 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.966391 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897}
I20260925 12:15:53.966730 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.960842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45769 (local address 127.0.98.3:50951)
0925 12:15:53.961153 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.961204 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.961850 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.962338 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.962343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.962354 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:53.963029 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.963038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.963945 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.963946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.964893 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.964895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.965275 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.965275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.965386 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.966002 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:53.966003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:53.966045 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:53.966081 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:53.966280 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:53.966286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:53.966289 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.966364 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.966476 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260925 12:15:53.967303 18416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45769
I20260925 12:15:53.967391 18416 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:53.967545 18416 heartbeater.cc:511] Master 127.0.98.62:45769 requested a full tablet report, sending...
I20260925 12:15:53.968024 17966 ts_manager.cc:194] Registered new tserver with Master: 1e4ef3210904436387eea114b81be3ab (127.0.98.3:34823)
I20260925 12:15:53.968430 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50951
I20260925 12:15:53.973657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:53.977703 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.975037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.62:45769)
0925 12:15:53.975104 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.975106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.975235 (+ 129us) server_negotiation.cc:407] Connection header received
0925 12:15:53.975297 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.975297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.975306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.975343 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:53.975460 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.975925 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.976716 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.976891 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.976962 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.977519 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:53.977520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:53.977520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:53.977524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:53.977537 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:53.977541 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:53.977595 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.977633 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.977650 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:15:53.979166 17966 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:53.980772 17966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:53.987068 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.983121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:41880)
0925 12:15:53.983391 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.983401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.983441 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.983819 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.983819 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.983821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.983871 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.983872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.985035 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.985036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.985909 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.985912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.986811 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.986812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.986872 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.986876 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.986938 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.986978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":30,"thread_start_us":59,"threads_started":1}
I20260925 12:15:53.987072 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.983372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.98.1:45115)
0925 12:15:53.983646 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.983650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.983661 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:53.983703 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.983705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.983719 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.983817 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.983948 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.984943 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.985981 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.986715 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.986796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.986886 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.986952 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.986989 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":78,"threads_started":1}
I20260925 12:15:53.987143 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.982838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.98.3:34823)
0925 12:15:53.983035 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.983040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.983050 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:53.983088 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.983090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.983107 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.983186 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.983344 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.984446 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.985912 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.986484 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.986590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.986689 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.987022 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.987061 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260925 12:15:53.987210 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.982733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34823 (local address 127.0.0.1:51008)
0925 12:15:53.982885 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.982916 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.982996 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.983192 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.983195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.983197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.983260 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.983262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.984566 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.984567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.985831 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.985833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.986724 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.986725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.986827 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.986841 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.986906 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.986953 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260925 12:15:53.987694 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.983318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.0.98.2:42729)
0925 12:15:53.983935 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:53.983939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:53.983948 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:53.983984 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:53.983985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:53.984001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:53.984075 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:53.984187 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.985050 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.986609 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.987322 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.987459 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.987553 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:53.987579 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 12:15:53.987619 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":85,"threads_started":1}
I20260925 12:15:53.987848 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:53.982965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.0.1:58384)
0925 12:15:53.983087 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:53.983097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:53.983144 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:53.984085 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:53.984086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:53.984088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:53.984120 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.984122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.985136 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.985137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.986530 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:53.986533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:53.987450 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:53.987451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:53.987520 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:53.987523 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:53.987563 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 12:15:53.987596 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260925 12:15:53.988044 18350 tablet_service.cc:1467] Processing CreateTablet for tablet 18b31e78766f4a23aa24eda4689ccffd (DEFAULT_TABLE table=test-table [id=340278c5642741158bd93909b256cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.988426 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b31e78766f4a23aa24eda4689ccffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.988528 18088 tablet_service.cc:1467] Processing CreateTablet for tablet 18b31e78766f4a23aa24eda4689ccffd (DEFAULT_TABLE table=test-table [id=340278c5642741158bd93909b256cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.988724 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 18b31e78766f4a23aa24eda4689ccffd (DEFAULT_TABLE table=test-table [id=340278c5642741158bd93909b256cf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:53.988902 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b31e78766f4a23aa24eda4689ccffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.989039 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b31e78766f4a23aa24eda4689ccffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:53.991674 18430 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: Bootstrap starting.
I20260925 12:15:53.991982 18431 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Bootstrap starting.
I20260925 12:15:53.992231 18432 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Bootstrap starting.
I20260925 12:15:53.992342 18430 tablet_bootstrap.cc:654] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.992690 18431 tablet_bootstrap.cc:654] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.992727 18430 log.cc:824] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.992985 18432 tablet_bootstrap.cc:654] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:53.993016 18431 log.cc:824] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.993310 18432 log.cc:824] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:53.993572 18430 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: No bootstrap required, opened a new log
I20260925 12:15:53.993678 18430 ts_tablet_manager.cc:1397] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 12:15:53.993683 18431 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: No bootstrap required, opened a new log
I20260925 12:15:53.993742 18431 ts_tablet_manager.cc:1397] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:53.994022 18432 tablet_bootstrap.cc:492] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: No bootstrap required, opened a new log
I20260925 12:15:53.994082 18432 ts_tablet_manager.cc:1397] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:53.995927 18430 raft_consensus.cc:348] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.996083 18430 raft_consensus.cc:374] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.996127 18430 raft_consensus.cc:729] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6661eb69ba4c0bb80fabb602bb7504, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.996255 18430 consensus_queue.cc:260] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.996392 18431 raft_consensus.cc:348] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.996589 18431 raft_consensus.cc:374] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.996641 18431 raft_consensus.cc:729] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4ef3210904436387eea114b81be3ab, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.996599 18432 raft_consensus.cc:348] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.996733 18432 raft_consensus.cc:374] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:53.996739 18430 ts_tablet_manager.cc:1428] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 12:15:53.996765 18432 raft_consensus.cc:729] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653b2af2e5044a45a7e8901c6d055a00, State: Initialized, Role: FOLLOWER
I20260925 12:15:53.996755 18431 consensus_queue.cc:260] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.996865 18154 heartbeater.cc:503] Master 127.0.98.62:45769 was elected leader, sending a full tablet report...
I20260925 12:15:53.996879 18432 consensus_queue.cc:260] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:53.997017 18431 ts_tablet_manager.cc:1428] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 12:15:53.997143 18416 heartbeater.cc:503] Master 127.0.98.62:45769 was elected leader, sending a full tablet report...
I20260925 12:15:53.997213 18285 heartbeater.cc:503] Master 127.0.98.62:45769 was elected leader, sending a full tablet report...
I20260925 12:15:53.997380 18432 ts_tablet_manager.cc:1428] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 12:15:54.059628 18286 tablet.cc:2369] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:54.112901 18436 raft_consensus.cc:482] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:54.113096 18436 raft_consensus.cc:504] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:54.113487 18436 leader_election.cc:288] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4ef3210904436387eea114b81be3ab (127.0.98.3:34823), 653b2af2e5044a45a7e8901c6d055a00 (127.0.98.2:42729)
I20260925 12:15:54.116565 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.113729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34823 (local address 127.0.98.1:50013)
0925 12:15:54.113792 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:54.113806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:54.113856 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:54.114031 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:54.114033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:54.114034 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:54.114104 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:54.114106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.114778 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.114780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:54.115643 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:54.115645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.116267 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.116268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:54.116347 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:54.116350 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:54.116427 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 12:15:54.116469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":7}
I20260925 12:15:54.116565 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.113802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50013 (local address 127.0.98.3:34823)
0925 12:15:54.113870 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:54.113873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:54.113887 (+ 14us) server_negotiation.cc:407] Connection header received
0925 12:15:54.113924 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:54.113925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:54.113934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:54.114012 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:54.114207 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.114680 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.115736 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.116163 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.116261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:54.116273 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:54.116437 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 12:15:54.116476 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:15:54.116920 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b31e78766f4a23aa24eda4689ccffd" candidate_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4ef3210904436387eea114b81be3ab" is_pre_election: true
I20260925 12:15:54.117089 18370 raft_consensus.cc:2454] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b6661eb69ba4c0bb80fabb602bb7504 in term 0.
I20260925 12:15:54.117244 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.113869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.98.1:38725)
0925 12:15:54.114090 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:54.114098 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:54.114186 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:54.114352 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:54.114353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:54.114355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:54.114438 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:54.114439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.115167 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.115168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:54.116076 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:54.116079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.116924 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.116925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:54.117006 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:54.117009 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:54.117115 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 12:15:54.117156 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":133,"threads_started":1}
I20260925 12:15:54.117237 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.113946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38725 (local address 127.0.98.2:42729)
0925 12:15:54.114023 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:54.114025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:54.114173 (+ 148us) server_negotiation.cc:407] Connection header received
0925 12:15:54.114248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:54.114249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:54.114269 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:54.114334 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:54.114511 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.115011 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.116215 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.116827 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.116916 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:54.116935 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:54.117126 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 12:15:54.117172 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 12:15:54.117333 18044 leader_election.cc:302] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4ef3210904436387eea114b81be3ab, 9b6661eb69ba4c0bb80fabb602bb7504; no voters:
I20260925 12:15:54.117456 18436 raft_consensus.cc:2781] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:54.117517 18436 raft_consensus.cc:482] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:54.117544 18436 raft_consensus.cc:3037] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:54.117717 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b31e78766f4a23aa24eda4689ccffd" candidate_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653b2af2e5044a45a7e8901c6d055a00" is_pre_election: true
I20260925 12:15:54.117870 18239 raft_consensus.cc:2454] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b6661eb69ba4c0bb80fabb602bb7504 in term 0.
I20260925 12:15:54.118443 18436 raft_consensus.cc:504] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:54.118571 18436 leader_election.cc:288] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4ef3210904436387eea114b81be3ab (127.0.98.3:34823), 653b2af2e5044a45a7e8901c6d055a00 (127.0.98.2:42729)
I20260925 12:15:54.118752 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b31e78766f4a23aa24eda4689ccffd" candidate_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653b2af2e5044a45a7e8901c6d055a00"
I20260925 12:15:54.118777 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b31e78766f4a23aa24eda4689ccffd" candidate_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4ef3210904436387eea114b81be3ab"
I20260925 12:15:54.118847 18370 raft_consensus.cc:3037] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:54.118852 18239 raft_consensus.cc:3037] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:54.119621 18370 raft_consensus.cc:2454] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b6661eb69ba4c0bb80fabb602bb7504 in term 1.
I20260925 12:15:54.119793 18044 leader_election.cc:302] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4ef3210904436387eea114b81be3ab, 9b6661eb69ba4c0bb80fabb602bb7504; no voters:
I20260925 12:15:54.119882 18436 raft_consensus.cc:2781] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:54.119911 18239 raft_consensus.cc:2454] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b6661eb69ba4c0bb80fabb602bb7504 in term 1.
I20260925 12:15:54.120024 18436 raft_consensus.cc:686] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [term 1 LEADER]: Becoming Leader. State: Replica: 9b6661eb69ba4c0bb80fabb602bb7504, State: Running, Role: LEADER
I20260925 12:15:54.120119 18436 consensus_queue.cc:237] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260925 12:15:54.121066 17964 catalog_manager.cc:5495] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b6661eb69ba4c0bb80fabb602bb7504 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b6661eb69ba4c0bb80fabb602bb7504" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } health_report { overall_health: HEALTHY } } }
I20260925 12:15:54.131323 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.128660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.98.1:45115)
0925 12:15:54.128717 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:54.128719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:54.128727 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:54.128766 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:54.128766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:54.128784 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:54.128870 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:54.129017 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.129523 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.130454 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:54.130676 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:54.130755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:54.130938 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:54.131041 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:54.131136 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:54.131187 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 12:15:54.131207 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:15:54.133193 18239 raft_consensus.cc:1261] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 1 FOLLOWER]: Refusing update from remote peer 9b6661eb69ba4c0bb80fabb602bb7504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:54.133217 18370 raft_consensus.cc:1261] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 1 FOLLOWER]: Refusing update from remote peer 9b6661eb69ba4c0bb80fabb602bb7504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:54.133579 18436 consensus_queue.cc:1035] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4ef3210904436387eea114b81be3ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:54.133711 18440 consensus_queue.cc:1035] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "653b2af2e5044a45a7e8901c6d055a00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:54.136569 18445 mvcc.cc:205] Tried to move back new op lower bound from 7333226717725691904 to 7333226717675692032. Current Snapshot: MvccSnapshot[applied={T|T < 7333226717725691904}]
I20260925 12:15:54.137647 18444 mvcc.cc:205] Tried to move back new op lower bound from 7333226717725691904 to 7333226717675692032. Current Snapshot: MvccSnapshot[applied={T|T < 7333226717725691904}]
I20260925 12:15:54.138675 17965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:54.138782 17965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:54.139581 17965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77828f972f754bd7b1106ea1f24038e0: Sending DeleteTablet for 3 replicas of tablet 18b31e78766f4a23aa24eda4689ccffd
I20260925 12:15:54.139982 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b31e78766f4a23aa24eda4689ccffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:54 UTC) from {username='slave'} at 127.0.0.1:41880
I20260925 12:15:54.140066 18350 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b31e78766f4a23aa24eda4689ccffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:54 UTC) from {username='slave'} at 127.0.0.1:51008
I20260925 12:15:54.140089 18219 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b31e78766f4a23aa24eda4689ccffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:54 UTC) from {username='slave'} at 127.0.0.1:58384
I20260925 12:15:54.140277 18453 tablet_replica.cc:330] T 18b31e78766f4a23aa24eda4689ccffd P 9b6661eb69ba4c0bb80fabb602bb7504: stopping tablet replica
I20260925 12:15:54.140295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18026
I20260925 12:15:54.140980 18454 tablet_replica.cc:330] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: stopping tablet replica
I20260925 12:15:54.141070 18454 raft_consensus.cc:2229] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:54.141142 18454 raft_consensus.cc:2258] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:54.141288 18455 tablet_replica.cc:330] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: stopping tablet replica
I20260925 12:15:54.141355 18455 raft_consensus.cc:2229] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:54.141424 18455 raft_consensus.cc:2258] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:54.141821 18454 ts_tablet_manager.cc:1905] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:54.143173 18455 ts_tablet_manager.cc:1905] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:54.143468 18454 ts_tablet_manager.cc:1918] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:54.143535 18454 log.cc:1196] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/18b31e78766f4a23aa24eda4689ccffd
I20260925 12:15:54.143836 18454 ts_tablet_manager.cc:1939] T 18b31e78766f4a23aa24eda4689ccffd P 1e4ef3210904436387eea114b81be3ab: Deleting consensus metadata
I20260925 12:15:54.144291 17953 catalog_manager.cc:4841] TS 1e4ef3210904436387eea114b81be3ab (127.0.98.3:34823): tablet 18b31e78766f4a23aa24eda4689ccffd (table test-table [id=340278c5642741158bd93909b256cf13]) successfully deleted
I20260925 12:15:54.145012 18455 ts_tablet_manager.cc:1918] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:54.145089 18455 log.cc:1196] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/18b31e78766f4a23aa24eda4689ccffd
I20260925 12:15:54.145339 18455 ts_tablet_manager.cc:1939] T 18b31e78766f4a23aa24eda4689ccffd P 653b2af2e5044a45a7e8901c6d055a00: Deleting consensus metadata
I20260925 12:15:54.146003 17953 catalog_manager.cc:4841] TS 653b2af2e5044a45a7e8901c6d055a00 (127.0.98.2:42729): tablet 18b31e78766f4a23aa24eda4689ccffd (table test-table [id=340278c5642741158bd93909b256cf13]) successfully deleted
W20260925 12:15:54.146279 17953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45115 (error 108)
I20260925 12:15:54.146559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18157
I20260925 12:15:54.146801 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:54.146661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:41900)
0925 12:15:54.146717 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:54.146760 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 12:15:54.146916 17953 catalog_manager.cc:4584] TS 9b6661eb69ba4c0bb80fabb602bb7504 (127.0.98.1:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18b31e78766f4a23aa24eda4689ccffd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
I20260925 12:15:54.152583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18288
I20260925 12:15:54.157306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 17935
2026-09-25T12:15:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:15:54 dist-test-slave-2hqq krb5kdc[18459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:15:54 dist-test-slave-2hqq krb5kdc[18459](info): set up 2 sockets
Sep 25 12:15:54 dist-test-slave-2hqq krb5kdc[18459](info): commencing operation
krb5kdc: starting...
W20260925 12:15:55.184204 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:15:55.303458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:55.383514 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:55.383761 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:55.383800 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:55.384977 18472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:55.385043 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:55.385067 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:55.385095 18472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:55.385126 18472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:55.386600 18472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:55.386863 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:55.387095 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:55.392067 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.392174 18472 server_base.cc:1034] running on GCE node
W20260925 12:15:55.392067 18480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:55.392156 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.392577 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:55.392936 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:55.394083 18472 hybrid_clock.cc:648] HybridClock initialized: now 1790338555394059 us; error 40 us; skew 500 ppm
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:55.402498 18472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:15:55.403023 18472 webserver.cc:460] Webserver started at http://127.0.98.62:34521/ using document root <none> and password file <none>
I20260925 12:15:55.403275 18472 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:55.403347 18472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:55.403492 18472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:55.404635 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "9c36b305ed5a4047971f3ebce85880d7"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "10bed97395290fae4277bf39aa315d34"
server_key_iv: "bb66e5b594d210ce319c204af0df293d"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.405011 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c36b305ed5a4047971f3ebce85880d7"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "10bed97395290fae4277bf39aa315d34"
server_key_iv: "bb66e5b594d210ce319c204af0df293d"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.406332 18472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:15:55.407073 18487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.407238 18472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.407344 18472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "9c36b305ed5a4047971f3ebce85880d7"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "10bed97395290fae4277bf39aa315d34"
server_key_iv: "bb66e5b594d210ce319c204af0df293d"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.407418 18472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:55.420765 18472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:55.422621 18472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:55.422837 18472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:55.426913 18472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40199
I20260925 12:15:55.426949 18539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40199 every 8 connection(s)
I20260925 12:15:55.427326 18472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:55.428045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18472
I20260925 12:15:55.428169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:55.428188 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:55.428407 392 external_mini_cluster.cc:1428] Setting key 3a94f359bf032584685d9513801b771e
I20260925 12:15:55.431627 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: Bootstrap starting.
I20260925 12:15:55.432294 18540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:55.432627 18540 log.cc:824] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:55.433424 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: No bootstrap required, opened a new log
I20260925 12:15:55.435731 18540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } }
I20260925 12:15:55.435910 18540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:55.435945 18540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c36b305ed5a4047971f3ebce85880d7, State: Initialized, Role: FOLLOWER
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:55.436070 18540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } }
I20260925 12:15:55.436160 18540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:55.436192 18540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:55.436236 18540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:55.436952 18540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } }
I20260925 12:15:55.437068 18540 leader_election.cc:302] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c36b305ed5a4047971f3ebce85880d7; no voters:
I20260925 12:15:55.437273 18540 leader_election.cc:288] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:55.437309 18545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:55.437422 18545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9c36b305ed5a4047971f3ebce85880d7, State: Running, Role: LEADER
I20260925 12:15:55.437565 18540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:55.437503 18545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } }
I20260925 12:15:55.439273 18546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c36b305ed5a4047971f3ebce85880d7. Latest consensus state: current_term: 1 leader_uuid: "9c36b305ed5a4047971f3ebce85880d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } } }
I20260925 12:15:55.439370 18546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:55.439555 18547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c36b305ed5a4047971f3ebce85880d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36b305ed5a4047971f3ebce85880d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40199 } } }
W20260925 12:15:55.439563 18560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 12:15:55.439623 18547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7 [sys.catalog]: This master's current role is: LEADER
W20260925 12:15:55.439692 18560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:15:55.439769 18562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:55.439936 18562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:55.440797 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.428945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.98.62:40199)
0925 12:15:55.429159 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:55.429170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:55.429202 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:15:55.429864 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:55.429873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:55.429936 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:55.430091 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:55.430674 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.432067 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.433244 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.433831 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.433956 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.436602 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:55.436606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:55.436607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:55.436618 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:55.439002 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.439463 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.439466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.439467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.439581 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.439873 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.439874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.439875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.440165 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:55.440311 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:55.440430 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 12:15:55.440605 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260925 12:15:55.441699 18562 catalog_manager.cc:1347] Generated new cluster ID: 263da004d4d84f72aa43e28a50b01073
I20260925 12:15:55.441759 18562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:15:55.451537 18562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:55.452319 18562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:55.469833 18562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: Generated new TSK 0
I20260925 12:15:55.470120 18562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:15:55.505481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:55.617031 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:55.617255 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:55.617281 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:55.618614 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:55.618660 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:55.618724 18566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:55.620393 18566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:55.620698 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:55.620975 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:55.626329 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.626430 18566 server_base.cc:1034] running on GCE node
W20260925 12:15:55.626515 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:55.626313 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.626905 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:55.627157 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:55.628328 18566 hybrid_clock.cc:648] HybridClock initialized: now 1790338555628312 us; error 27 us; skew 500 ppm
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:55.631001 18566 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:15:55.631384 18566 webserver.cc:460] Webserver started at http://127.0.98.1:41197/ using document root <none> and password file <none>
I20260925 12:15:55.631589 18566 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:55.631632 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:55.631719 18566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:55.632795 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f13565da3634ddcad99089207d8d8fc"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "2c46dd764519d69978b3902c86779ab6"
server_key_iv: "335703c5338e99bf656f066b5446c3fe"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.633093 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f13565da3634ddcad99089207d8d8fc"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "2c46dd764519d69978b3902c86779ab6"
server_key_iv: "335703c5338e99bf656f066b5446c3fe"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.634349 18566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:15:55.635051 18581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.635259 18566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:55.635356 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "6f13565da3634ddcad99089207d8d8fc"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "2c46dd764519d69978b3902c86779ab6"
server_key_iv: "335703c5338e99bf656f066b5446c3fe"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.635430 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:55.646891 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:55.649348 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:55.649524 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:55.649770 18566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:55.650087 18566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:55.650143 18566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.650178 18566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:55.650204 18566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.655800 18566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46139
I20260925 12:15:55.655880 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46139 every 8 connection(s)
I20260925 12:15:55.656212 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:55.661250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18566
I20260925 12:15:55.661339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:55.661607 392 external_mini_cluster.cc:1428] Setting key 066cf75c6f33fcb35299ba06ac5db09c
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:55.666400 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.657093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56925 (local address 127.0.98.62:40199)
0925 12:15:55.657158 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:55.657160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:55.657350 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:15:55.658340 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:55.658341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:55.658354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:55.658410 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:55.659051 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.659575 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.660698 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.660927 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.661027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.663527 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:55.663528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:55.663528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:55.663536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:55.665001 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.665407 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.665409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.665409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.665471 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.665746 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.665747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.665748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.665863 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:55.665934 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:55.666221 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 12:15:55.666307 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 12:15:55.666605 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.657085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40199 (local address 127.0.98.1:56925)
0925 12:15:55.657228 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:55.657264 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:55.658225 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:55.658503 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:55.658507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:55.658756 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:55.658944 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.658948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.659685 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.659686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.660539 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.660541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.661038 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.661039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.661176 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.661784 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:55.661794 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:55.663407 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:55.665102 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.665109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.665111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.665330 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.665539 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.665540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.665541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.665667 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.665940 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:55.665947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:55.666089 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 12:15:55.666199 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 12:15:55.666356 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260925 12:15:55.667109 18695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40199
I20260925 12:15:55.667200 18695 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:55.667397 18695 heartbeater.cc:511] Master 127.0.98.62:40199 requested a full tablet report, sending...
I20260925 12:15:55.667974 18504 ts_manager.cc:194] Registered new tserver with Master: 6f13565da3634ddcad99089207d8d8fc (127.0.98.1:46139)
I20260925 12:15:55.669009 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56925
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:15:55.688117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:55.769927 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:55.770141 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:55.770164 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:55.771279 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:55.771317 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:55.771345 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:55.772886 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:55.773075 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:55.773247 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:55.778137 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.778230 18700 server_base.cc:1034] running on GCE node
W20260925 12:15:55.778106 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:55.778306 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.778565 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:55.778806 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:55.779973 18700 hybrid_clock.cc:648] HybridClock initialized: now 1790338555779962 us; error 37 us; skew 500 ppm
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:55.782584 18700 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:15:55.783000 18700 webserver.cc:460] Webserver started at http://127.0.98.2:33733/ using document root <none> and password file <none>
I20260925 12:15:55.783231 18700 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:55.783298 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:55.783449 18700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:55.784715 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "79c7ed33215a49deb050c59b2e34f558"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "04ac3e0949f2eb00ac084f1c23d06254"
server_key_iv: "59cae1f28780fe5de4f2d92b20d96de6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.785077 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79c7ed33215a49deb050c59b2e34f558"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "04ac3e0949f2eb00ac084f1c23d06254"
server_key_iv: "59cae1f28780fe5de4f2d92b20d96de6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.786378 18700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:55.787124 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.787289 18700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:55.787390 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "79c7ed33215a49deb050c59b2e34f558"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "04ac3e0949f2eb00ac084f1c23d06254"
server_key_iv: "59cae1f28780fe5de4f2d92b20d96de6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.787470 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:55.805730 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:55.808172 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:55.808342 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:55.808612 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:55.808934 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:55.808993 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.809033 18700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:55.809082 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.815104 18700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35697
I20260925 12:15:55.815177 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35697 every 8 connection(s)
I20260925 12:15:55.815505 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:55.823159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18700
I20260925 12:15:55.823292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:55.823531 392 external_mini_cluster.cc:1428] Setting key 2e86142363d8c12a8622653609fa487e
I20260925 12:15:55.825402 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.816276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42677 (local address 127.0.98.62:40199)
0925 12:15:55.816366 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:55.816368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:55.816580 (+ 212us) server_negotiation.cc:407] Connection header received
0925 12:15:55.817585 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:55.817586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:55.817596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:55.817651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:55.818270 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.818815 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.819876 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.820064 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.820117 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.822265 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:55.822267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:55.822267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:55.822274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:55.823889 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.824375 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.824376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.824376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.824463 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.824754 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.824756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.824756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.824878 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:55.824960 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:55.825225 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 12:15:55.825313 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 12:15:55.825784 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.816239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40199 (local address 127.0.98.2:42677)
0925 12:15:55.816415 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:55.816451 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:55.817490 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:55.817707 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:55.817710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:55.817994 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:55.818193 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.818197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.818958 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.818959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.819799 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.819801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.820139 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.820140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.820240 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.820754 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:55.820763 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:55.822183 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:55.824009 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.824016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.824018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.824279 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.824572 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.824573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.824574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.824682 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.824951 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:55.824957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:55.825094 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 12:15:55.825205 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 12:15:55.825361 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260925 12:15:55.826309 18829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40199
I20260925 12:15:55.826402 18829 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:55.826613 18829 heartbeater.cc:511] Master 127.0.98.62:40199 requested a full tablet report, sending...
I20260925 12:15:55.827040 18504 ts_manager.cc:194] Registered new tserver with Master: 79c7ed33215a49deb050c59b2e34f558 (127.0.98.2:35697)
I20260925 12:15:55.827419 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42677
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:15:55.849174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:55.930444 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:55.930662 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:55.930687 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:55.931823 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:55.931859 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:55.931885 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:55.933506 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:55.933713 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:55.933895 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:55.939088 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:55.939085 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.939255 18834 server_base.cc:1034] running on GCE node
W20260925 12:15:55.939292 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:55.939510 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:55.939779 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:55.940929 18834 hybrid_clock.cc:648] HybridClock initialized: now 1790338555940914 us; error 26 us; skew 500 ppm
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:55.943392 18834 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:15:55.943819 18834 webserver.cc:460] Webserver started at http://127.0.98.3:42389/ using document root <none> and password file <none>
I20260925 12:15:55.944046 18834 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:55.944092 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:55.944213 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:55.945308 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "9ff451591e3cfb5dbb50e6123f1a85d7"
server_key_iv: "5c90418e92bd3691c2bc5422527cbfa6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.945655 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "9ff451591e3cfb5dbb50e6123f1a85d7"
server_key_iv: "5c90418e92bd3691c2bc5422527cbfa6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.946979 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:15:55.947736 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.947921 18834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:55.947991 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d"
format_stamp: "Formatted at 2026-09-25 12:15:55 on dist-test-slave-2hqq"
server_key: "9ff451591e3cfb5dbb50e6123f1a85d7"
server_key_iv: "5c90418e92bd3691c2bc5422527cbfa6"
server_key_version: "encryptionkey@0"
I20260925 12:15:55.948076 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:55.968806 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:55.971068 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:55.971213 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:55.971472 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:55.971817 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:55.971853 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.971896 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:55.971921 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:55.978029 18834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36941
I20260925 12:15:55.978070 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36941 every 8 connection(s)
I20260925 12:15:55.978385 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:55.984225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 18834
I20260925 12:15:55.984335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:55.984581 392 external_mini_cluster.cc:1428] Setting key b5de7b733416d177917acc381530affd
Sep 25 12:15:55 dist-test-slave-2hqq krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:55.988173 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.979095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39503 (local address 127.0.98.62:40199)
0925 12:15:55.979145 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:55.979147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:55.979364 (+ 217us) server_negotiation.cc:407] Connection header received
0925 12:15:55.980275 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:55.980276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:55.980286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:55.980339 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:55.981008 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.981501 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.982468 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.982657 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.982712 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.985691 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:55.985693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:55.985694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:55.985701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:55.986909 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.987318 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.987319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.987319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.987372 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:55.987601 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:55.987601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:55.987602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:55.987708 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:55.987785 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:55.988034 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 12:15:55.988110 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:15:55.988471 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.979078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40199 (local address 127.0.98.3:39503)
0925 12:15:55.979239 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:55.979277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:55.980183 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:55.980397 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:55.980401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:55.980729 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:55.980930 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.980934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.981583 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.981583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.982392 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:55.982394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.982733 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:55.982734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:55.982844 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:55.983562 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:55.983570 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:55.985440 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:55.986996 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.987002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.987004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.987247 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.987432 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:55.987432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:55.987433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:55.987539 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:55.987788 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:55.987793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:55.987911 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 12:15:55.987999 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 12:15:55.988180 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260925 12:15:55.988996 18963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40199
I20260925 12:15:55.989104 18963 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:55.989300 18963 heartbeater.cc:511] Master 127.0.98.62:40199 requested a full tablet report, sending...
I20260925 12:15:55.989718 18504 ts_manager.cc:194] Registered new tserver with Master: 9fb5da0a8e8a43e0bcbec98dd47bf85d (127.0.98.3:36941)
I20260925 12:15:55.990165 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39503
I20260925 12:15:55.996783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:56.004190 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:55.998234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.0.98.62:40199)
0925 12:15:55.998332 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:55.998334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:55.998449 (+ 115us) server_negotiation.cc:407] Connection header received
0925 12:15:55.998483 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:55.998484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:55.998494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:55.998553 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:55.999088 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:55.999563 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.000257 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.000427 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.000475 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.001861 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.001862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.001862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:56.001867 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.003018 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:56.003365 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:56.003366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:56.003366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:56.003414 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:56.003643 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:56.003644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:56.003644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:56.003764 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.003842 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.003999 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.004046 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 12:15:56.005859 18504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:56.007386 18504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:56.014142 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.009742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35697 (local address 127.0.0.1:56576)
0925 12:15:56.010045 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.010109 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.010165 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.010460 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.010461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.010466 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.010560 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.010563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011582 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.011583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.012906 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.012909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.013819 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.013896 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.013909 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.013982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.014040 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":24,"thread_start_us":79,"threads_started":1}
I20260925 12:15:56.014153 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.009871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.0.98.2:35697)
0925 12:15:56.010158 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.010162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.010172 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:56.010268 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.010271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.010287 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.010358 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.010651 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011481 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.012998 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.013744 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.013882 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.014011 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.014052 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260925 12:15:56.014329 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.010169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.3:36941)
0925 12:15:56.010357 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.010361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.010372 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:56.010410 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.010412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.010426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.010498 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.010697 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011806 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013161 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.013797 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013865 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.013923 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.014186 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.014231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260925 12:15:56.014479 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.009871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.98.1:46139)
0925 12:15:56.010063 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.010068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.010080 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:15:56.010201 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.010205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.010223 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.010294 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.010650 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011813 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013033 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.013872 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013936 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.013998 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.014335 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.014366 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260925 12:15:56.014273 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.009742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.0.1:54142)
0925 12:15:56.009915 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.009977 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.010040 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.010457 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.010460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.010463 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.010556 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.010559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011948 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.011949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.012925 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.012927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.014033 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.014034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.014092 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.014096 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.014146 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.014191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":51,"thread_start_us":68,"threads_started":1}
I20260925 12:15:56.014149 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.009945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36941 (local address 127.0.0.1:36362)
0925 12:15:56.010214 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.010224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.010264 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.010577 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.010578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.010580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.010623 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.010624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.011940 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.011941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.013082 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.013084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.013954 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.013955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.014010 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.014013 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.014056 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.014087 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":25,"thread_start_us":70,"threads_started":1}
I20260925 12:15:56.015301 18763 tablet_service.cc:1467] Processing CreateTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb (DEFAULT_TABLE table=test-table [id=9f42f739c1344112bab933f4fce16f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.015311 18629 tablet_service.cc:1467] Processing CreateTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb (DEFAULT_TABLE table=test-table [id=9f42f739c1344112bab933f4fce16f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.015486 18897 tablet_service.cc:1467] Processing CreateTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb (DEFAULT_TABLE table=test-table [id=9f42f739c1344112bab933f4fce16f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.015676 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.015676 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.015764 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.018425 18980 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Bootstrap starting.
I20260925 12:15:56.018985 18982 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Bootstrap starting.
I20260925 12:15:56.019263 18981 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Bootstrap starting.
I20260925 12:15:56.019366 18980 tablet_bootstrap.cc:654] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.019639 18982 tablet_bootstrap.cc:654] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.019883 18981 tablet_bootstrap.cc:654] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.019990 18980 log.cc:824] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.020260 18982 log.cc:824] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.020529 18981 log.cc:824] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.020876 18980 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: No bootstrap required, opened a new log
I20260925 12:15:56.020975 18980 ts_tablet_manager.cc:1397] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:15:56.021054 18982 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: No bootstrap required, opened a new log
I20260925 12:15:56.021137 18982 ts_tablet_manager.cc:1397] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:56.021382 18981 tablet_bootstrap.cc:492] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: No bootstrap required, opened a new log
I20260925 12:15:56.021450 18981 ts_tablet_manager.cc:1397] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:56.022661 18980 raft_consensus.cc:348] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.022869 18980 raft_consensus.cc:374] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.022915 18980 raft_consensus.cc:729] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c7ed33215a49deb050c59b2e34f558, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.023031 18980 consensus_queue.cc:260] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.023147 18982 raft_consensus.cc:348] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.023321 18982 raft_consensus.cc:374] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.023342 18980 ts_tablet_manager.cc:1428] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 12:15:56.023365 18982 raft_consensus.cc:729] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb5da0a8e8a43e0bcbec98dd47bf85d, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.023469 18982 consensus_queue.cc:260] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.023589 18829 heartbeater.cc:503] Master 127.0.98.62:40199 was elected leader, sending a full tablet report...
I20260925 12:15:56.023506 18981 raft_consensus.cc:348] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.023672 18981 raft_consensus.cc:374] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.023720 18981 raft_consensus.cc:729] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f13565da3634ddcad99089207d8d8fc, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.023737 18982 ts_tablet_manager.cc:1428] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:56.023830 18981 consensus_queue.cc:260] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.023870 18963 heartbeater.cc:503] Master 127.0.98.62:40199 was elected leader, sending a full tablet report...
I20260925 12:15:56.024112 18695 heartbeater.cc:503] Master 127.0.98.62:40199 was elected leader, sending a full tablet report...
I20260925 12:15:56.024322 18981 ts_tablet_manager.cc:1428] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 12:15:56.066171 18830 tablet.cc:2369] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:15:56.156976 18696 tablet.cc:2369] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:56.167702 18988 raft_consensus.cc:482] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:56.167824 18988 raft_consensus.cc:504] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.168207 18988 leader_election.cc:288] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79c7ed33215a49deb050c59b2e34f558 (127.0.98.2:35697), 9fb5da0a8e8a43e0bcbec98dd47bf85d (127.0.98.3:36941)
I20260925 12:15:56.171226 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.168578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41733 (local address 127.0.98.3:36941)
0925 12:15:56.168633 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.168635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.168646 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:56.168744 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.168745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.168753 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.168824 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.168980 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.169500 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170511 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.170879 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170962 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.170972 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.171116 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.171159 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:15:56.171272 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.168530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56287 (local address 127.0.98.2:35697)
0925 12:15:56.168576 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.168578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.168657 (+ 79us) server_negotiation.cc:407] Connection header received
0925 12:15:56.168783 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.168784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.168795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.168863 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.169017 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.169502 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170517 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.170895 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170989 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.170998 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.171165 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.171210 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:15:56.171273 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.168419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36941 (local address 127.0.98.1:41733)
0925 12:15:56.168560 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.168574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.168661 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.168829 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.168830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.168832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.168907 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.168909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.169593 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.169594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.170418 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.170420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.170962 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.171039 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.171043 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.171111 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.171150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":19,"thread_start_us":92,"threads_started":1}
I20260925 12:15:56.171289 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.168419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35697 (local address 127.0.98.1:56287)
0925 12:15:56.168589 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.168601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.168666 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.168867 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.168868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.168869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.168944 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.168945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.169597 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.169598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.170420 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.170422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.170992 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.170993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.171072 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.171075 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.171157 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.171197 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":53,"thread_start_us":187,"threads_started":1}
I20260925 12:15:56.171782 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e4f62cd27c4c21b2f3d67cfd7e71cb" candidate_uuid: "6f13565da3634ddcad99089207d8d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c7ed33215a49deb050c59b2e34f558" is_pre_election: true
I20260925 12:15:56.171780 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e4f62cd27c4c21b2f3d67cfd7e71cb" candidate_uuid: "6f13565da3634ddcad99089207d8d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" is_pre_election: true
I20260925 12:15:56.171941 18783 raft_consensus.cc:2454] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f13565da3634ddcad99089207d8d8fc in term 0.
I20260925 12:15:56.171945 18917 raft_consensus.cc:2454] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f13565da3634ddcad99089207d8d8fc in term 0.
I20260925 12:15:56.172170 18584 leader_election.cc:302] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f13565da3634ddcad99089207d8d8fc, 79c7ed33215a49deb050c59b2e34f558; no voters:
I20260925 12:15:56.172305 18988 raft_consensus.cc:2781] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:56.172355 18988 raft_consensus.cc:482] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:56.172371 18988 raft_consensus.cc:3037] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:56.173075 18988 raft_consensus.cc:504] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.173224 18988 leader_election.cc:288] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [CANDIDATE]: Term 1 election: Requested vote from peers 79c7ed33215a49deb050c59b2e34f558 (127.0.98.2:35697), 9fb5da0a8e8a43e0bcbec98dd47bf85d (127.0.98.3:36941)
I20260925 12:15:56.173370 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e4f62cd27c4c21b2f3d67cfd7e71cb" candidate_uuid: "6f13565da3634ddcad99089207d8d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c7ed33215a49deb050c59b2e34f558"
I20260925 12:15:56.173417 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e4f62cd27c4c21b2f3d67cfd7e71cb" candidate_uuid: "6f13565da3634ddcad99089207d8d8fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d"
I20260925 12:15:56.173453 18783 raft_consensus.cc:3037] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:56.173498 18917 raft_consensus.cc:3037] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:56.174198 18917 raft_consensus.cc:2454] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f13565da3634ddcad99089207d8d8fc in term 1.
I20260925 12:15:56.174322 18783 raft_consensus.cc:2454] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f13565da3634ddcad99089207d8d8fc in term 1.
I20260925 12:15:56.174419 18583 leader_election.cc:302] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f13565da3634ddcad99089207d8d8fc, 9fb5da0a8e8a43e0bcbec98dd47bf85d; no voters:
I20260925 12:15:56.174530 18988 raft_consensus.cc:2781] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:56.174721 18988 raft_consensus.cc:686] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 1 LEADER]: Becoming Leader. State: Replica: 6f13565da3634ddcad99089207d8d8fc, State: Running, Role: LEADER
I20260925 12:15:56.174930 18988 consensus_queue.cc:237] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } }
I20260925 12:15:56.175853 18503 catalog_manager.cc:5495] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f13565da3634ddcad99089207d8d8fc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f13565da3634ddcad99089207d8d8fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f13565da3634ddcad99089207d8d8fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 } health_report { overall_health: UNKNOWN } } }
I20260925 12:15:56.197710 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.194793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.0.98.1:46139)
0925 12:15:56.194840 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.194842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.194850 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:56.194902 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.194903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.194922 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.194984 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:56.195108 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.195583 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.196586 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.196880 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.196951 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.197208 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:56.197328 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:56.197436 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.197616 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.197628 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:15:56.200107 18783 raft_consensus.cc:1261] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 1 FOLLOWER]: Refusing update from remote peer 6f13565da3634ddcad99089207d8d8fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:56.200121 18917 raft_consensus.cc:1261] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 1 FOLLOWER]: Refusing update from remote peer 6f13565da3634ddcad99089207d8d8fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:56.200516 18991 consensus_queue.cc:1035] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb5da0a8e8a43e0bcbec98dd47bf85d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:56.200665 18988 consensus_queue.cc:1035] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c7ed33215a49deb050c59b2e34f558" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:56.203534 18995 mvcc.cc:205] Tried to move back new op lower bound from 7333226726192209920 to 7333226726093135872. Current Snapshot: MvccSnapshot[applied={T|T < 7333226726192209920}]
I20260925 12:15:56.204411 18996 mvcc.cc:205] Tried to move back new op lower bound from 7333226726192209920 to 7333226726093135872. Current Snapshot: MvccSnapshot[applied={T|T < 7333226726192209920 or (T in {7333226726192209920})}]
I20260925 12:15:56.205483 18503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:56.205585 18503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:56.206492 18503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c36b305ed5a4047971f3ebce85880d7: Sending DeleteTablet for 3 replicas of tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb
I20260925 12:15:56.206869 18629 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54142
I20260925 12:15:56.206928 18763 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260925 12:15:56.207108 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36362
I20260925 12:15:56.207134 19003 tablet_replica.cc:330] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: stopping tablet replica
I20260925 12:15:56.207237 19003 raft_consensus.cc:2229] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:15:56.207314 19005 tablet_replica.cc:330] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: stopping tablet replica
I20260925 12:15:56.207325 19003 raft_consensus.cc:2258] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:56.207471 19005 raft_consensus.cc:2229] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:56.207505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18566
I20260925 12:15:56.207576 19005 raft_consensus.cc:2258] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:56.207739 19003 ts_tablet_manager.cc:1905] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 6f13565da3634ddcad99089207d8d8fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:56.210151 19005 ts_tablet_manager.cc:1905] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:56.211941 19005 ts_tablet_manager.cc:1918] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:56.212034 19005 log.cc:1196] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/96e4f62cd27c4c21b2f3d67cfd7e71cb
I20260925 12:15:56.212347 19005 ts_tablet_manager.cc:1939] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 9fb5da0a8e8a43e0bcbec98dd47bf85d: Deleting consensus metadata
W20260925 12:15:56.212738 18489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46139 (error 108)
I20260925 12:15:56.213003 18489 catalog_manager.cc:4841] TS 9fb5da0a8e8a43e0bcbec98dd47bf85d (127.0.98.3:36941): tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb (table test-table [id=9f42f739c1344112bab933f4fce16f1d]) successfully deleted
W20260925 12:15:56.213336 18978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:15:56.213180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.0.1:54162)
0925 12:15:56.213213 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.213223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.213269 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.213302 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46139: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 12:15:56.213460 18489 catalog_manager.cc:4584] TS 6f13565da3634ddcad99089207d8d8fc (127.0.98.1:46139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96e4f62cd27c4c21b2f3d67cfd7e71cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46139: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 12:15:56.213522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18700
I20260925 12:15:56.213531 19004 tablet_replica.cc:330] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558: stopping tablet replica
I20260925 12:15:56.213639 19004 raft_consensus.cc:2229] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:56.213743 19004 raft_consensus.cc:2258] T 96e4f62cd27c4c21b2f3d67cfd7e71cb P 79c7ed33215a49deb050c59b2e34f558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:56.218608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18834
I20260925 12:15:56.218647 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.218491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35697 (local address 127.0.0.1:56590)
0925 12:15:56.218561 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.218610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 12:15:56.223114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 18472
2026-09-25T12:15:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T12:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:56Z Disabled control of system clock
I20260925 12:15:56.250531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:56.330989 19010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:56.331214 19010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:56.331234 19010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:56.332305 19010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:56.332340 19010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:56.332352 19010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:56.332365 19010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:56.332376 19010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:56.333866 19010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:56.334062 19010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:56.334371 19010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:56.339231 19018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.339269 19015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.339293 19016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:56.339556 19010 server_base.cc:1034] running on GCE node
I20260925 12:15:56.339803 19010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:56.340085 19010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:56.341245 19010 hybrid_clock.cc:648] HybridClock initialized: now 1790338556341233 us; error 34 us; skew 500 ppm
I20260925 12:15:56.342500 19010 webserver.cc:460] Webserver started at http://127.0.98.62:44285/ using document root <none> and password file <none>
I20260925 12:15:56.342756 19010 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:56.342826 19010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:56.342960 19010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:56.344045 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "1589682b9bd14cc7bd80f24bcb649101"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "e3ac165d57205589681bc550b4386cc0"
server_key_iv: "2cdb21d6b17ed5cffeace94b1b49fb63"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.344420 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "1589682b9bd14cc7bd80f24bcb649101"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "e3ac165d57205589681bc550b4386cc0"
server_key_iv: "2cdb21d6b17ed5cffeace94b1b49fb63"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.345721 19010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:56.346464 19024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.346654 19010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:56.346752 19010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "1589682b9bd14cc7bd80f24bcb649101"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "e3ac165d57205589681bc550b4386cc0"
server_key_iv: "2cdb21d6b17ed5cffeace94b1b49fb63"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.346836 19010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:56.376968 19010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:56.377367 19010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:56.377538 19010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:56.382082 19010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41845
I20260925 12:15:56.382119 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41845 every 8 connection(s)
I20260925 12:15:56.382519 19010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:56.383463 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.385075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19010
I20260925 12:15:56.385192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:56.385403 392 external_mini_cluster.cc:1428] Setting key c9863c777d0a7fa34231ef7a9e1246ea
I20260925 12:15:56.386407 19077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: Bootstrap starting.
I20260925 12:15:56.387072 19077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.387416 19077 log.cc:824] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.388164 19077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: No bootstrap required, opened a new log
I20260925 12:15:56.390188 19077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260925 12:15:56.390379 19077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.390434 19077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1589682b9bd14cc7bd80f24bcb649101, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.390563 19077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260925 12:15:56.390628 19077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:56.390650 19077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:56.390715 19077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:56.391203 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.385986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.98.62:41845)
0925 12:15:56.386181 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.386189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.386213 (+ 24us) server_negotiation.cc:407] Connection header received
0925 12:15:56.386685 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.386692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.386760 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.386879 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:56.387163 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.388260 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.389134 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.389541 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.389672 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.390402 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.390406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.390406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:56.390668 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.390712 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:56.390726 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.390786 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.390922 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.391011 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260925 12:15:56.391381 19077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260925 12:15:56.391489 19077 leader_election.cc:302] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1589682b9bd14cc7bd80f24bcb649101; no voters:
I20260925 12:15:56.391650 19077 leader_election.cc:288] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:56.391680 19082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:56.391844 19082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [term 1 LEADER]: Becoming Leader. State: Replica: 1589682b9bd14cc7bd80f24bcb649101, State: Running, Role: LEADER
I20260925 12:15:56.391911 19077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:56.391928 19082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } }
I20260925 12:15:56.392333 19083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1589682b9bd14cc7bd80f24bcb649101. Latest consensus state: current_term: 1 leader_uuid: "1589682b9bd14cc7bd80f24bcb649101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } } }
I20260925 12:15:56.392421 19083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:56.392421 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1589682b9bd14cc7bd80f24bcb649101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1589682b9bd14cc7bd80f24bcb649101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41845 } } }
I20260925 12:15:56.392671 19087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:56.392688 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:56.393222 19087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:56.394793 19087 catalog_manager.cc:1347] Generated new cluster ID: 24896726533b411096535c7c339189b3
I20260925 12:15:56.394863 19087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:15:56.419819 19087 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:56.420392 19087 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:56.430689 19087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: Generated new TSK 0
I20260925 12:15:56.430930 19087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:15:56.433374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--builtin_ntp_servers=127.0.98.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:56.530918 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:56.531118 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:56.531139 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:56.532271 19101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:56.532317 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:56.532343 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:56.533717 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:56.533955 19101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:56.534247 19101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:56.539124 19109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.539126 19107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.539129 19106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:56.539477 19101 server_base.cc:1034] running on GCE node
I20260925 12:15:56.539674 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:56.539916 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:56.541090 19101 hybrid_clock.cc:648] HybridClock initialized: now 1790338556541066 us; error 38 us; skew 500 ppm
I20260925 12:15:56.542395 19101 webserver.cc:460] Webserver started at http://127.0.98.1:45057/ using document root <none> and password file <none>
I20260925 12:15:56.542644 19101 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:56.542716 19101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:56.542941 19101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:56.544009 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "0374d2a402c3415aad50aa08de58171d"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "25a0e881e485e5f61eaca05678f4218e"
server_key_iv: "aa41cde63c424ab40baa456cf8604a76"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.544405 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0374d2a402c3415aad50aa08de58171d"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "25a0e881e485e5f61eaca05678f4218e"
server_key_iv: "aa41cde63c424ab40baa456cf8604a76"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.545899 19101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:56.546674 19115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.546859 19101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:56.546957 19101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "0374d2a402c3415aad50aa08de58171d"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "25a0e881e485e5f61eaca05678f4218e"
server_key_iv: "aa41cde63c424ab40baa456cf8604a76"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.547039 19101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:56.565032 19101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:56.565418 19101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:56.565590 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:56.565842 19101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:56.566174 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:56.566241 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.566303 19101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:56.566334 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.572088 19101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37945
I20260925 12:15:56.572132 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37945 every 8 connection(s)
I20260925 12:15:56.572463 19101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:56.577445 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.573219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59605 (local address 127.0.98.62:41845)
0925 12:15:56.573299 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.573301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.573487 (+ 186us) server_negotiation.cc:407] Connection header received
0925 12:15:56.573975 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.573976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.573990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.574046 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:56.574685 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.575194 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.576113 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.576285 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.576333 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.577238 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.577239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.577239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:56.577243 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.577253 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:56.577256 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.577295 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.577368 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.577386 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 12:15:56.577641 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.573173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.1:59605)
0925 12:15:56.573360 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.573395 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.573887 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.574099 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.574102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.574110 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:56.574606 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.574610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.575278 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.575279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.576040 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.576042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.576376 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.576377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.576532 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.577117 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:56.577118 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:56.577146 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:56.577167 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:56.577295 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:56.577297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:56.577299 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.577359 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.577433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260925 12:15:56.578152 19229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260925 12:15:56.578243 19229 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:56.578387 19229 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260925 12:15:56.578940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19101
I20260925 12:15:56.578946 19041 ts_manager.cc:194] Registered new tserver with Master: 0374d2a402c3415aad50aa08de58171d (127.0.98.1:37945)
I20260925 12:15:56.579041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:56.579252 392 external_mini_cluster.cc:1428] Setting key 0f8ac2abceafcfdc34868a7c52de0ba4
I20260925 12:15:56.579828 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59605
I20260925 12:15:56.580240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--builtin_ntp_servers=127.0.98.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:56.661388 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:56.661635 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:56.661670 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:56.662799 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:56.662863 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:56.662907 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:56.664230 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:56.664511 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:56.664808 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:56.669778 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.669785 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:56.669996 19232 server_base.cc:1034] running on GCE node
W20260925 12:15:56.670090 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:56.670377 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:56.670615 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:56.671775 19232 hybrid_clock.cc:648] HybridClock initialized: now 1790338556671758 us; error 38 us; skew 500 ppm
I20260925 12:15:56.673179 19232 webserver.cc:460] Webserver started at http://127.0.98.2:35239/ using document root <none> and password file <none>
I20260925 12:15:56.673421 19232 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:56.673494 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:56.673651 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:56.674803 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "a81fc59dd84847199565f69a766e61bd"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "9bbbe28b217a9776d8f3e5432f27fb55"
server_key_iv: "f4f1fe1e2e2cb5a351a82c958848dc10"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.675168 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a81fc59dd84847199565f69a766e61bd"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "9bbbe28b217a9776d8f3e5432f27fb55"
server_key_iv: "f4f1fe1e2e2cb5a351a82c958848dc10"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.676429 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:15:56.677201 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.677357 19232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:56.677443 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "a81fc59dd84847199565f69a766e61bd"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "9bbbe28b217a9776d8f3e5432f27fb55"
server_key_iv: "f4f1fe1e2e2cb5a351a82c958848dc10"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.677523 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:56.697773 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:56.698117 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:56.698266 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:56.698522 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:56.698864 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:56.698906 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.698930 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:56.698942 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.704684 19232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43075
I20260925 12:15:56.704800 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43075 every 8 connection(s)
I20260925 12:15:56.705142 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:15:56.710316 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.705969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55543 (local address 127.0.98.62:41845)
0925 12:15:56.706055 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.706057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.706209 (+ 152us) server_negotiation.cc:407] Connection header received
0925 12:15:56.706738 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.706739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.706749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.706803 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:56.707418 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.707901 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.708964 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.709151 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.709203 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.710104 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.710105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.710105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:56.710109 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.710120 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:56.710123 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.710175 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.710238 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.710258 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 12:15:56.710536 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.705904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.2:55543)
0925 12:15:56.706090 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.706124 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.706649 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.706857 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.706860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.706869 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:56.707352 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.707356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.707978 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.707978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.708892 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.708895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.709226 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.709227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.709332 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.709969 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:56.709969 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:56.709999 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:56.710022 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:56.710177 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:56.710180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:56.710182 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.710230 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.710308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":97,"threads_started":1}
I20260925 12:15:56.711108 19360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260925 12:15:56.711212 19360 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:56.711395 19360 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260925 12:15:56.711797 19041 ts_manager.cc:194] Registered new tserver with Master: a81fc59dd84847199565f69a766e61bd (127.0.98.2:43075)
I20260925 12:15:56.712179 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55543
I20260925 12:15:56.715058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19232
I20260925 12:15:56.715183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:56.715404 392 external_mini_cluster.cc:1428] Setting key b191c8a10b50bd5cf2d9cf69050dd17f
I20260925 12:15:56.716303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--builtin_ntp_servers=127.0.98.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:56.798367 19363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:56.798611 19363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:56.798645 19363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:56.799837 19363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:56.799909 19363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:56.799952 19363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:56.801362 19363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:56.801661 19363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:56.801985 19363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:15:56.807288 19363 server_base.cc:1034] running on GCE node
W20260925 12:15:56.807224 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.807224 19368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:56.807226 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:56.807683 19363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:56.807901 19363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:56.809039 19363 hybrid_clock.cc:648] HybridClock initialized: now 1790338556809020 us; error 37 us; skew 500 ppm
I20260925 12:15:56.810322 19363 webserver.cc:460] Webserver started at http://127.0.98.3:41291/ using document root <none> and password file <none>
I20260925 12:15:56.810616 19363 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:56.810689 19363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:56.810837 19363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:56.811892 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "48b1b25a19f84dbda917a23ef67fc1dc"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "1f0a5a9cb71965424a9e88a5c1b0714b"
server_key_iv: "546dc9cf50f7b03ead1b7e09467bc830"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.812261 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48b1b25a19f84dbda917a23ef67fc1dc"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "1f0a5a9cb71965424a9e88a5c1b0714b"
server_key_iv: "546dc9cf50f7b03ead1b7e09467bc830"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.813683 19363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:15:56.814491 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.814664 19363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:56.814759 19363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "48b1b25a19f84dbda917a23ef67fc1dc"
format_stamp: "Formatted at 2026-09-25 12:15:56 on dist-test-slave-2hqq"
server_key: "1f0a5a9cb71965424a9e88a5c1b0714b"
server_key_iv: "546dc9cf50f7b03ead1b7e09467bc830"
server_key_version: "encryptionkey@0"
I20260925 12:15:56.814841 19363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:56.869306 19363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:56.869673 19363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:56.869835 19363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:56.870085 19363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:56.870455 19363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:56.870512 19363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.870553 19363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:56.870579 19363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:56.876518 19363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46829
I20260925 12:15:56.876657 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46829 every 8 connection(s)
I20260925 12:15:56.876963 19363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:56.881552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19363
I20260925 12:15:56.881654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:56.881847 392 external_mini_cluster.cc:1428] Setting key 352070b69d334f6860b4a28feb9a5b61
I20260925 12:15:56.882409 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.877701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60351 (local address 127.0.98.62:41845)
0925 12:15:56.877745 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.877747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.877986 (+ 239us) server_negotiation.cc:407] Connection header received
0925 12:15:56.878472 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.878474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.878493 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.878580 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:56.879255 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.879762 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.880838 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.881072 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.881125 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.882067 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.882069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.882069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:56.882095 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.882111 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:56.882116 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.882176 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.882298 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.882324 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:15:56.882745 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.877668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41845 (local address 127.0.98.3:60351)
0925 12:15:56.877858 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.877898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.878361 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.878674 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.878678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.878687 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:56.879183 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.879187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.879875 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.879876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.880756 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.880758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.881151 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.881152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.881299 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.881886 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:56.881887 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:56.881949 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:56.881979 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:56.882200 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:56.882207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:56.882210 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.882288 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.882411 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 12:15:56.883172 19491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41845
I20260925 12:15:56.883251 19491 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:56.883437 19491 heartbeater.cc:511] Master 127.0.98.62:41845 requested a full tablet report, sending...
I20260925 12:15:56.883826 19041 ts_manager.cc:194] Registered new tserver with Master: 48b1b25a19f84dbda917a23ef67fc1dc (127.0.98.3:46829)
I20260925 12:15:56.884205 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60351
I20260925 12:15:56.893980 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:56.898296 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.895429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.0.98.62:41845)
0925 12:15:56.895504 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.895506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.895573 (+ 67us) server_negotiation.cc:407] Connection header received
0925 12:15:56.895628 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.895629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.895637 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.895678 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:56.895824 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.896296 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.897107 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.897286 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.897338 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.897991 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:56.897992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:56.897993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:56.897997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:56.898010 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:56.898014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:56.898098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.898161 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.898171 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 12:15:56.899780 19041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:56.901417 19041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:56.907809 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.903637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46829 (local address 127.0.0.1:59156)
0925 12:15:56.903745 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.903785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.903845 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.904370 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.904371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.904373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.904425 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.904426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905868 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.905869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.906861 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.906862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.907556 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.907557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.907602 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.907604 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.907650 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.907700 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260925 12:15:56.907833 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.903743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.1:37945)
0925 12:15:56.903937 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.903942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.903951 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:56.903995 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.903997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.904008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.904117 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.904406 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905299 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.906663 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.907400 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.907473 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.907571 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.907702 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.907745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260925 12:15:56.907810 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.903649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:52994)
0925 12:15:56.903792 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.903803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.903856 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.904236 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.904239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.904241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.904312 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.904315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905380 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.905381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.906577 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.906579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.907490 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.907492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.907584 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.907599 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.907674 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.907708 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260925 12:15:56.908260 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.903836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43075 (local address 127.0.0.1:48234)
0925 12:15:56.904380 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:56.904390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:56.904434 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:56.904731 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:56.904732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:56.904734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:56.904774 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.904775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905988 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.905989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.907009 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:56.907012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.908005 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.908006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:56.908087 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.908092 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:15:56.908141 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:15:56.908188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":36,"threads_started":1}
I20260925 12:15:56.908347 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.903939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.3:46829)
0925 12:15:56.904117 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.904122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.904134 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:15:56.904171 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.904173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.904189 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.904532 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.904596 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905771 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.906942 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.907461 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.908060 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.908165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.908194 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.908243 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260925 12:15:56.908656 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:56.904224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.2:43075)
0925 12:15:56.904568 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:56.904572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:56.904582 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:56.904619 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:56.904621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:56.904637 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:56.904715 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:56.904856 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.905909 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.907105 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:56.907815 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:56.907882 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:56.907960 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:56.908522 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 12:15:56.908570 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":80,"threads_started":1}
I20260925 12:15:56.908886 19163 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8794670e894d729e6b414eddff4dc6 (DEFAULT_TABLE table=test-table [id=3e16cbe0b59b4ed8b36ac4424b5c9a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.908905 19425 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8794670e894d729e6b414eddff4dc6 (DEFAULT_TABLE table=test-table [id=3e16cbe0b59b4ed8b36ac4424b5c9a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.909260 19163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8794670e894d729e6b414eddff4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.909260 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8794670e894d729e6b414eddff4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.909513 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8794670e894d729e6b414eddff4dc6 (DEFAULT_TABLE table=test-table [id=3e16cbe0b59b4ed8b36ac4424b5c9a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:56.909832 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8794670e894d729e6b414eddff4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:56.912544 19505 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Bootstrap starting.
I20260925 12:15:56.912890 19507 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Bootstrap starting.
I20260925 12:15:56.913251 19505 tablet_bootstrap.cc:654] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.913475 19507 tablet_bootstrap.cc:654] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.913583 19505 log.cc:824] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.913803 19507 log.cc:824] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.914108 19506 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Bootstrap starting.
I20260925 12:15:56.914366 19505 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: No bootstrap required, opened a new log
I20260925 12:15:56.914456 19505 ts_tablet_manager.cc:1397] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:56.914682 19506 tablet_bootstrap.cc:654] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:56.914997 19506 log.cc:824] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:56.915699 19506 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: No bootstrap required, opened a new log
I20260925 12:15:56.915777 19506 ts_tablet_manager.cc:1397] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:56.916589 19505 raft_consensus.cc:348] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.916731 19505 raft_consensus.cc:374] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.916754 19505 raft_consensus.cc:729] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81fc59dd84847199565f69a766e61bd, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.916889 19505 consensus_queue.cc:260] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.917201 19360 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
I20260925 12:15:56.917454 19505 ts_tablet_manager.cc:1428] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:56.918087 19507 tablet_bootstrap.cc:492] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: No bootstrap required, opened a new log
I20260925 12:15:56.918161 19507 ts_tablet_manager.cc:1397] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 12:15:56.918069 19506 raft_consensus.cc:348] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.918252 19506 raft_consensus.cc:374] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.918296 19506 raft_consensus.cc:729] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b1b25a19f84dbda917a23ef67fc1dc, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.918430 19506 consensus_queue.cc:260] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.918721 19506 ts_tablet_manager.cc:1428] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:56.918777 19491 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
I20260925 12:15:56.920632 19507 raft_consensus.cc:348] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.920773 19507 raft_consensus.cc:374] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:56.920818 19507 raft_consensus.cc:729] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374d2a402c3415aad50aa08de58171d, State: Initialized, Role: FOLLOWER
I20260925 12:15:56.920934 19507 consensus_queue.cc:260] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:56.921175 19507 ts_tablet_manager.cc:1428] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:56.921245 19229 heartbeater.cc:503] Master 127.0.98.62:41845 was elected leader, sending a full tablet report...
W20260925 12:15:56.955727 19361 tablet.cc:2369] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:15:57.073179 19230 tablet.cc:2369] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:57.096760 19513 raft_consensus.cc:482] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:57.096927 19513 raft_consensus.cc:504] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:57.097342 19513 leader_election.cc:288] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b1b25a19f84dbda917a23ef67fc1dc (127.0.98.3:46829), a81fc59dd84847199565f69a766e61bd (127.0.98.2:43075)
I20260925 12:15:57.100313 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.097647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40419 (local address 127.0.98.3:46829)
0925 12:15:57.097731 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:57.097733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:57.097861 (+ 128us) server_negotiation.cc:407] Connection header received
0925 12:15:57.097896 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:57.097897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:57.097908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:57.097967 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:57.098165 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.098664 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.099612 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.099967 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.100050 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:57.100060 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:57.100198 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 12:15:57.100236 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 12:15:57.100358 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.097906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51087 (local address 127.0.98.2:43075)
0925 12:15:57.097981 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:57.097982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:57.098002 (+ 20us) server_negotiation.cc:407] Connection header received
0925 12:15:57.098029 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:57.098030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:57.098043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:57.098092 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:57.098253 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.098738 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.099653 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.100008 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.100089 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:57.100098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:57.100246 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 12:15:57.100283 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 12:15:57.100373 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.097622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46829 (local address 127.0.98.1:40419)
0925 12:15:57.097729 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:57.097741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:57.097807 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:57.098018 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:57.098019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:57.098021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:57.098082 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:57.098084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.098753 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.098755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:57.099509 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:57.099511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.100051 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.100052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:57.100125 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:57.100128 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:57.100194 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:15:57.100233 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260925 12:15:57.100375 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.097810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43075 (local address 127.0.98.1:51087)
0925 12:15:57.097900 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:57.097910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:57.097978 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:57.098096 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:57.098097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:57.098099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:57.098179 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:57.098181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.098825 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.098826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:57.099564 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:57.099566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.100091 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.100092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:57.100162 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:57.100165 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:57.100226 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 12:15:57.100264 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":11,"thread_start_us":54,"threads_started":1}
I20260925 12:15:57.100816 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8794670e894d729e6b414eddff4dc6" candidate_uuid: "0374d2a402c3415aad50aa08de58171d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a81fc59dd84847199565f69a766e61bd" is_pre_election: true
I20260925 12:15:57.100821 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8794670e894d729e6b414eddff4dc6" candidate_uuid: "0374d2a402c3415aad50aa08de58171d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" is_pre_election: true
I20260925 12:15:57.100986 19314 raft_consensus.cc:2454] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374d2a402c3415aad50aa08de58171d in term 0.
I20260925 12:15:57.100998 19445 raft_consensus.cc:2454] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374d2a402c3415aad50aa08de58171d in term 0.
I20260925 12:15:57.101236 19118 leader_election.cc:302] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374d2a402c3415aad50aa08de58171d, 48b1b25a19f84dbda917a23ef67fc1dc; no voters:
I20260925 12:15:57.101364 19513 raft_consensus.cc:2781] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:57.101418 19513 raft_consensus.cc:482] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:57.101442 19513 raft_consensus.cc:3037] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:57.102144 19513 raft_consensus.cc:504] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:57.102272 19513 leader_election.cc:288] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [CANDIDATE]: Term 1 election: Requested vote from peers 48b1b25a19f84dbda917a23ef67fc1dc (127.0.98.3:46829), a81fc59dd84847199565f69a766e61bd (127.0.98.2:43075)
I20260925 12:15:57.102434 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8794670e894d729e6b414eddff4dc6" candidate_uuid: "0374d2a402c3415aad50aa08de58171d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b1b25a19f84dbda917a23ef67fc1dc"
I20260925 12:15:57.102545 19445 raft_consensus.cc:3037] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:57.102538 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8794670e894d729e6b414eddff4dc6" candidate_uuid: "0374d2a402c3415aad50aa08de58171d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a81fc59dd84847199565f69a766e61bd"
I20260925 12:15:57.102687 19314 raft_consensus.cc:3037] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:57.103387 19445 raft_consensus.cc:2454] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374d2a402c3415aad50aa08de58171d in term 1.
I20260925 12:15:57.103461 19314 raft_consensus.cc:2454] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374d2a402c3415aad50aa08de58171d in term 1.
I20260925 12:15:57.103552 19118 leader_election.cc:302] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374d2a402c3415aad50aa08de58171d, 48b1b25a19f84dbda917a23ef67fc1dc; no voters:
I20260925 12:15:57.103657 19513 raft_consensus.cc:2781] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:57.103818 19513 raft_consensus.cc:686] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [term 1 LEADER]: Becoming Leader. State: Replica: 0374d2a402c3415aad50aa08de58171d, State: Running, Role: LEADER
I20260925 12:15:57.103935 19513 consensus_queue.cc:237] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } }
I20260925 12:15:57.104849 19041 catalog_manager.cc:5495] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374d2a402c3415aad50aa08de58171d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0374d2a402c3415aad50aa08de58171d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374d2a402c3415aad50aa08de58171d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } health_report { overall_health: HEALTHY } } }
W20260925 12:15:57.127552 19492 tablet.cc:2369] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:57.133783 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.131500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.98.1:37945)
0925 12:15:57.131559 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:57.131561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:57.131569 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:57.131609 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:57.131611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:57.131637 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:57.131701 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:57.131815 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.132284 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.133093 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:57.133286 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:57.133340 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:57.133467 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:57.133551 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:57.133627 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:57.133710 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:15:57.133720 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 12:15:57.135735 19314 raft_consensus.cc:1261] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 1 FOLLOWER]: Refusing update from remote peer 0374d2a402c3415aad50aa08de58171d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:57.135735 19445 raft_consensus.cc:1261] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 1 FOLLOWER]: Refusing update from remote peer 0374d2a402c3415aad50aa08de58171d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:57.136093 19513 consensus_queue.cc:1035] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81fc59dd84847199565f69a766e61bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:57.136238 19516 consensus_queue.cc:1035] T 4f8794670e894d729e6b414eddff4dc6 P 0374d2a402c3415aad50aa08de58171d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b1b25a19f84dbda917a23ef67fc1dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:57.139747 19521 mvcc.cc:205] Tried to move back new op lower bound from 7333226730024914944 to 7333226729898319872. Current Snapshot: MvccSnapshot[applied={T|T < 7333226730024914944}]
I20260925 12:15:57.140249 19520 mvcc.cc:205] Tried to move back new op lower bound from 7333226730024914944 to 7333226729898319872. Current Snapshot: MvccSnapshot[applied={T|T < 7333226730024914944 or (T in {7333226730024914944})}]
I20260925 12:15:57.141569 19040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:57.141682 19040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:57.142419 19040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1589682b9bd14cc7bd80f24bcb649101: Sending DeleteTablet for 3 replicas of tablet 4f8794670e894d729e6b414eddff4dc6
I20260925 12:15:57.142762 19163 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8794670e894d729e6b414eddff4dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20260925 12:15:57.142839 19425 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8794670e894d729e6b414eddff4dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:57 UTC) from {username='slave'} at 127.0.0.1:59156
I20260925 12:15:57.143035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19101
I20260925 12:15:57.143062 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8794670e894d729e6b414eddff4dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:57 UTC) from {username='slave'} at 127.0.0.1:48234
I20260925 12:15:57.143775 19531 tablet_replica.cc:330] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: stopping tablet replica
I20260925 12:15:57.143903 19531 raft_consensus.cc:2229] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:57.144048 19531 raft_consensus.cc:2258] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:57.144464 19531 ts_tablet_manager.cc:1905] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:57.146093 19531 ts_tablet_manager.cc:1918] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:57.146162 19531 log.cc:1196] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/4f8794670e894d729e6b414eddff4dc6
I20260925 12:15:57.146415 19531 ts_tablet_manager.cc:1939] T 4f8794670e894d729e6b414eddff4dc6 P a81fc59dd84847199565f69a766e61bd: Deleting consensus metadata
I20260925 12:15:57.146904 19027 catalog_manager.cc:4841] TS a81fc59dd84847199565f69a766e61bd (127.0.98.2:43075): tablet 4f8794670e894d729e6b414eddff4dc6 (table test-table [id=3e16cbe0b59b4ed8b36ac4424b5c9a5d]) successfully deleted
W20260925 12:15:57.148918 19028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37945 (error 108)
I20260925 12:15:57.149446 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:57.149315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:53020)
0925 12:15:57.149370 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:57.149418 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:15:57.149438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19232
I20260925 12:15:57.149451 19530 tablet_replica.cc:330] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: stopping tablet replica
I20260925 12:15:57.149566 19530 raft_consensus.cc:2229] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:57.149680 19530 raft_consensus.cc:2258] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 12:15:57.149755 19028 catalog_manager.cc:4584] TS 0374d2a402c3415aad50aa08de58171d (127.0.98.1:37945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f8794670e894d729e6b414eddff4dc6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
I20260925 12:15:57.152895 19530 ts_tablet_manager.cc:1905] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:57.154934 19530 ts_tablet_manager.cc:1918] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:57.155012 19530 log.cc:1196] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/4f8794670e894d729e6b414eddff4dc6
I20260925 12:15:57.155269 19530 ts_tablet_manager.cc:1939] T 4f8794670e894d729e6b414eddff4dc6 P 48b1b25a19f84dbda917a23ef67fc1dc: Deleting consensus metadata
I20260925 12:15:57.156026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19363
I20260925 12:15:57.156025 19027 catalog_manager.cc:4841] TS 48b1b25a19f84dbda917a23ef67fc1dc (127.0.98.3:46829): tablet 4f8794670e894d729e6b414eddff4dc6 (table test-table [id=3e16cbe0b59b4ed8b36ac4424b5c9a5d]) successfully deleted
I20260925 12:15:57.161150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19010
2026-09-25T12:15:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (936 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:15:57 dist-test-slave-2hqq krb5kdc[19535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:15:57 dist-test-slave-2hqq krb5kdc[19535](info): set up 2 sockets
Sep 25 12:15:57 dist-test-slave-2hqq krb5kdc[19535](info): commencing operation
krb5kdc: starting...
W20260925 12:15:58.196462 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:15:58.319514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:58.401652 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:58.401858 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:58.401880 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:58.403076 19548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:58.403122 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:58.403137 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:58.403151 19548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:58.403162 19548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:58.404775 19548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:58.405031 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:58.405288 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:58.410135 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:58.410142 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.410392 19548 server_base.cc:1034] running on GCE node
W20260925 12:15:58.410142 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.410673 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:58.410995 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:58.412154 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790338558412133 us; error 39 us; skew 500 ppm
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:58.414760 19548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:15:58.415261 19548 webserver.cc:460] Webserver started at http://127.0.98.62:36093/ using document root <none> and password file <none>
I20260925 12:15:58.415509 19548 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:58.415556 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:58.415709 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:58.416885 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "3be4015ba4a44d8eb40e333b85e33657"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "d33567cac78c8e3c266253d932e61cc6"
server_key_iv: "31feb195b46e9b0fe3128f3f2ba61ce7"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.417255 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "3be4015ba4a44d8eb40e333b85e33657"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "d33567cac78c8e3c266253d932e61cc6"
server_key_iv: "31feb195b46e9b0fe3128f3f2ba61ce7"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.418605 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:15:58.419432 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.419610 19548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:58.419677 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "3be4015ba4a44d8eb40e333b85e33657"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "d33567cac78c8e3c266253d932e61cc6"
server_key_iv: "31feb195b46e9b0fe3128f3f2ba61ce7"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.419766 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:58.428205 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:58.430235 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:58.430382 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:58.434468 19548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38945
I20260925 12:15:58.434526 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38945 every 8 connection(s)
I20260925 12:15:58.434871 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:58.435783 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:58.438203 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: Bootstrap starting.
I20260925 12:15:58.438802 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:58.439095 19616 log.cc:824] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:58.439822 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: No bootstrap required, opened a new log
I20260925 12:15:58.441583 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } }
I20260925 12:15:58.441727 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:58.441751 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be4015ba4a44d8eb40e333b85e33657, State: Initialized, Role: FOLLOWER
I20260925 12:15:58.441908 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } }
I20260925 12:15:58.441979 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:58.442000 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:58.442071 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:58.442597 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } }
I20260925 12:15:58.442713 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be4015ba4a44d8eb40e333b85e33657; no voters:
I20260925 12:15:58.442895 19616 leader_election.cc:288] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:58.442941 19619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:58.443121 19619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [term 1 LEADER]: Becoming Leader. State: Replica: 3be4015ba4a44d8eb40e333b85e33657, State: Running, Role: LEADER
I20260925 12:15:58.443171 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:58.443288 19619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } }
I20260925 12:15:58.443699 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be4015ba4a44d8eb40e333b85e33657. Latest consensus state: current_term: 1 leader_uuid: "3be4015ba4a44d8eb40e333b85e33657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } } }
I20260925 12:15:58.443819 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:58.443856 19620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be4015ba4a44d8eb40e333b85e33657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be4015ba4a44d8eb40e333b85e33657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38945 } } }
I20260925 12:15:58.443949 19620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:58.444089 19628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:58.444396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19548
I20260925 12:15:58.444525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:58.444769 392 external_mini_cluster.cc:1428] Setting key f91f4de0eda6a4160c4879f318cc36ec
I20260925 12:15:58.444805 19628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:58.446280 19628 catalog_manager.cc:1347] Generated new cluster ID: 03e6b6793caa490b95b89b946b6eee5f
I20260925 12:15:58.446341 19628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:58.454866 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.445394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.98.62:38945)
0925 12:15:58.445551 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.445560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.445595 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:15:58.445716 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.445722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.445793 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.445956 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:58.446710 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.447660 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.448446 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.448946 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.449071 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.451440 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:58.451444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:58.451445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:58.451476 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:58.453204 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.453586 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.453590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.453590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.453700 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.453932 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.453933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.453933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.454179 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:58.454355 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.454453 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.454700 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260925 12:15:58.463052 19628 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:58.463779 19628 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:58.476076 19628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: Generated new TSK 0
I20260925 12:15:58.476300 19628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 12:15:58.505759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:58.587836 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:58.588052 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:58.588075 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:58.589278 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:58.589319 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:58.589346 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:58.590981 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:58.591226 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:58.591456 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:58.596668 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:58.596674 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.596722 19642 server_base.cc:1034] running on GCE node
W20260925 12:15:58.596674 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.597193 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:58.597400 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:58.598556 19642 hybrid_clock.cc:648] HybridClock initialized: now 1790338558598546 us; error 31 us; skew 500 ppm
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:58.601251 19642 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 12:15:58.601686 19642 webserver.cc:460] Webserver started at http://127.0.98.1:34997/ using document root <none> and password file <none>
I20260925 12:15:58.601943 19642 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:58.602015 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:58.602160 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:58.603231 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "e740730a9e5247a1a727dc556406c5a4"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "7cef86dd415fa1557746c8cace084f85"
server_key_iv: "0732c64bc3d637ea896d3e279cc68dda"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.603591 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e740730a9e5247a1a727dc556406c5a4"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "7cef86dd415fa1557746c8cace084f85"
server_key_iv: "0732c64bc3d637ea896d3e279cc68dda"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.604971 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:15:58.605662 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.605834 19642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:15:58.605952 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "e740730a9e5247a1a727dc556406c5a4"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "7cef86dd415fa1557746c8cace084f85"
server_key_iv: "0732c64bc3d637ea896d3e279cc68dda"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.606026 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:58.620100 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:58.622041 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:58.622221 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:58.622480 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:58.622782 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:58.622838 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.622877 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:58.622908 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.628620 19642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41231
I20260925 12:15:58.628691 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41231 every 8 connection(s)
I20260925 12:15:58.629021 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:58.630596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19642
I20260925 12:15:58.630685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:58.630910 392 external_mini_cluster.cc:1428] Setting key 56c5acf76b758b7f5d6ce2e0e42265af
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:58.639906 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.629716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45661 (local address 127.0.98.62:38945)
0925 12:15:58.629783 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.629785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.630025 (+ 240us) server_negotiation.cc:407] Connection header received
0925 12:15:58.631128 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.631130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.631144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.631228 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:58.631955 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.632624 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.634206 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.634552 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.634627 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.636855 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:58.636857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:58.636857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:58.636866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:58.638509 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.638906 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.638907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.638908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.638966 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.639197 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.639199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.639199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.639361 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:58.639453 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.639717 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.639812 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 12:15:58.639971 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.629703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38945 (local address 127.0.98.1:45661)
0925 12:15:58.629893 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.629940 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.631020 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.631298 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.631302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.631641 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:58.631840 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.631845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.632708 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.632709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.634114 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.634117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.634648 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.634649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.634763 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.635270 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:58.635278 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:58.636761 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:58.638597 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.638604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.638606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.638822 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.639035 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.639036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.639036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.639129 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.639456 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:58.639462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:58.639579 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.639666 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.639785 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 12:15:58.640545 19771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38945
I20260925 12:15:58.640650 19771 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:58.640838 19771 heartbeater.cc:511] Master 127.0.98.62:38945 requested a full tablet report, sending...
I20260925 12:15:58.641556 19580 ts_manager.cc:194] Registered new tserver with Master: e740730a9e5247a1a727dc556406c5a4 (127.0.98.1:41231)
I20260925 12:15:58.642168 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 12:15:58.659618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:58.742020 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:58.742231 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:58.742255 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:58.743559 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:58.743600 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:58.743647 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:58.745335 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:58.745582 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:58.745831 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:58.751034 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:58.751065 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:58.751039 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.751235 19776 server_base.cc:1034] running on GCE node
I20260925 12:15:58.751494 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:58.751740 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:58.752935 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790338558752921 us; error 44 us; skew 500 ppm
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:58.755491 19776 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 12:15:58.755890 19776 webserver.cc:460] Webserver started at http://127.0.98.2:37971/ using document root <none> and password file <none>
I20260925 12:15:58.756069 19776 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:58.756110 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:58.756201 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:58.757278 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "be1010f133cd4db7a58931ad57461365"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "434aa517232f99d4e36e49714f7fc0e9"
server_key_iv: "053d21c677a1271787b422ed5bf792bd"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.757596 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be1010f133cd4db7a58931ad57461365"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "434aa517232f99d4e36e49714f7fc0e9"
server_key_iv: "053d21c677a1271787b422ed5bf792bd"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.759007 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:15:58.759804 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.759975 19776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:58.760063 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "be1010f133cd4db7a58931ad57461365"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "434aa517232f99d4e36e49714f7fc0e9"
server_key_iv: "053d21c677a1271787b422ed5bf792bd"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.760121 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:58.778702 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:58.781152 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:58.781270 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:58.781455 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:58.781713 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:58.781742 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.781764 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:58.781776 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.787235 19776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34275
I20260925 12:15:58.787336 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34275 every 8 connection(s)
I20260925 12:15:58.787629 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:58.794801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19776
I20260925 12:15:58.794926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:58.795184 392 external_mini_cluster.cc:1428] Setting key 69608f3d0905b3fec944635b6555eac3
I20260925 12:15:58.797458 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.788350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49549 (local address 127.0.98.62:38945)
0925 12:15:58.788436 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.788438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.788686 (+ 248us) server_negotiation.cc:407] Connection header received
0925 12:15:58.789664 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.789666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.789678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.789728 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:58.790322 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.790818 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.791840 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.792036 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.792092 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.794243 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:58.794244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:58.794245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:58.794251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:58.795752 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.796265 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.796266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.796266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.796319 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.796695 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.796696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.796696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.796818 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:58.796876 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.797211 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.797332 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 12:15:58.797814 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.788299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38945 (local address 127.0.98.2:49549)
0925 12:15:58.788534 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.788580 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.789563 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.789793 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.789797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.790059 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:58.790250 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.790254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.790897 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.790897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.791756 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.791758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.792131 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.792133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.792242 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.792765 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:58.792773 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:58.794151 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:58.795845 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.795851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.795853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.796190 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.796383 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.796384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.796384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.796582 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.796884 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:58.796890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:58.797062 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.797169 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.797341 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260925 12:15:58.798352 19905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38945
I20260925 12:15:58.798463 19905 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:58.798658 19905 heartbeater.cc:511] Master 127.0.98.62:38945 requested a full tablet report, sending...
I20260925 12:15:58.799075 19580 ts_manager.cc:194] Registered new tserver with Master: be1010f133cd4db7a58931ad57461365 (127.0.98.2:34275)
I20260925 12:15:58.799454 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49549
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 12:15:58.820847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:15:58.904130 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:58.904330 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:58.904353 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:58.905560 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:58.905602 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:58.905629 19910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:58.907088 19910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:58.907337 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:58.907564 19910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:58.912609 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:58.912611 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.912851 19910 server_base.cc:1034] running on GCE node
W20260925 12:15:58.912621 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:58.913161 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:58.913417 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:58.914580 19910 hybrid_clock.cc:648] HybridClock initialized: now 1790338558914560 us; error 35 us; skew 500 ppm
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:15:58.917066 19910 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 12:15:58.917536 19910 webserver.cc:460] Webserver started at http://127.0.98.3:33349/ using document root <none> and password file <none>
I20260925 12:15:58.917773 19910 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:58.917840 19910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:58.917989 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:58.919054 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "954fa6bc29204339945cfdecda1df1be"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "1b04850b24b0fd2d0ee7a1ee1942f3ba"
server_key_iv: "d6727306fc1ae862cebf47dd7a9034aa"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.919441 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "954fa6bc29204339945cfdecda1df1be"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "1b04850b24b0fd2d0ee7a1ee1942f3ba"
server_key_iv: "d6727306fc1ae862cebf47dd7a9034aa"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.920979 19910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:15:58.921720 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.921892 19910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:58.922010 19910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "954fa6bc29204339945cfdecda1df1be"
format_stamp: "Formatted at 2026-09-25 12:15:58 on dist-test-slave-2hqq"
server_key: "1b04850b24b0fd2d0ee7a1ee1942f3ba"
server_key_iv: "d6727306fc1ae862cebf47dd7a9034aa"
server_key_version: "encryptionkey@0"
I20260925 12:15:58.922088 19910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:58.941360 19910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:58.943581 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:58.943804 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:58.944104 19910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:58.944435 19910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:58.944471 19910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.944551 19910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:58.944586 19910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:58.950595 19910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38373
I20260925 12:15:58.950619 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38373 every 8 connection(s)
I20260925 12:15:58.950971 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:58.955945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 19910
I20260925 12:15:58.956028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:58.956240 392 external_mini_cluster.cc:1428] Setting key 312eaf210e9ad70724cd8bc43368d990
Sep 25 12:15:58 dist-test-slave-2hqq krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:15:58.961004 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.951713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57599 (local address 127.0.98.62:38945)
0925 12:15:58.951755 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.951757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.951995 (+ 238us) server_negotiation.cc:407] Connection header received
0925 12:15:58.953045 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.953046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.953061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.953121 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:58.953785 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.954371 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.955453 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.955675 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.955782 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.958326 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:58.958328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:58.958328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:58.958336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:58.959545 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.959971 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.959973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.959973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.960037 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.960311 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.960312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.960312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.960433 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:58.960524 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.960823 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.960907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:15:58.961207 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.951691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38945 (local address 127.0.98.3:57599)
0925 12:15:58.951875 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.951911 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.952928 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.953198 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.953201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.953463 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:58.953676 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.953680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.954470 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.954472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.955342 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.955345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.955774 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.955775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.955912 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.956539 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:15:58.956549 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:58.958228 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:58.959643 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.959650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.959652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.959880 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.960119 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:15:58.960120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:15:58.960121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:15:58.960236 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:15:58.960521 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:58.960529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:58.960685 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.960800 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.960966 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260925 12:15:58.961709 20039 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38945
I20260925 12:15:58.961813 20039 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:58.961962 20039 heartbeater.cc:511] Master 127.0.98.62:38945 requested a full tablet report, sending...
I20260925 12:15:58.962334 19580 ts_manager.cc:194] Registered new tserver with Master: 954fa6bc29204339945cfdecda1df1be (127.0.98.3:38373)
I20260925 12:15:58.962718 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57599
I20260925 12:15:58.968269 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:58.975592 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.969707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.0.98.62:38945)
0925 12:15:58.969768 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.969771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.969918 (+ 147us) server_negotiation.cc:407] Connection header received
0925 12:15:58.969990 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.969991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.970000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.970045 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:58.970575 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.971336 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.972045 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.972231 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.972279 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.973508 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:58.973509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:58.973509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:15:58.973515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:58.974663 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.974965 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.974966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.974966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.975014 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:15:58.975177 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:15:58.975178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:15:58.975178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:15:58.975315 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:58.975388 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.975471 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.975526 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 12:15:58.977336 19580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:58.978997 19580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:58.986066 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34275 (local address 127.0.0.1:59872)
0925 12:15:58.981513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.981556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.981629 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.982153 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.982156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.982158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:58.982229 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.982232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983609 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.983610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.984860 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.984863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985714 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.985795 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.985808 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.985905 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.985971 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260925 12:15:58.986145 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.0.98.1:41231)
0925 12:15:58.981879 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.981885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.982070 (+ 185us) server_negotiation.cc:407] Connection header received
0925 12:15:58.982319 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.982322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.982340 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.982413 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:58.982494 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983537 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.984939 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985771 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985850 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.985905 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.985988 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.986030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260925 12:15:58.986281 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.3:38373)
0925 12:15:58.981948 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.981951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.981961 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:58.982057 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.982060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.982077 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.982161 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:58.982375 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983106 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985036 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985771 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985856 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.985905 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.986156 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.986203 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":111,"threads_started":1}
I20260925 12:15:58.986068 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41231 (local address 127.0.0.1:46880)
0925 12:15:58.982003 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.982014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.982134 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.982411 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.982412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.982415 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:58.982449 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.982450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983657 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.983658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.984846 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.984849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985853 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.985924 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.985927 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.985976 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.986005 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":92,"thread_start_us":71,"threads_started":1}
I20260925 12:15:58.986738 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.98.2:34275)
0925 12:15:58.981892 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:58.981898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:58.981910 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:15:58.981951 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:58.981953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:58.981970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:58.982310 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:58.982389 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983480 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.984990 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985605 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.985777 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:58.986611 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 12:15:58.986660 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260925 12:15:58.986505 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:58.981367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38373 (local address 127.0.0.1:51892)
0925 12:15:58.981894 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:58.981905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:58.981952 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:58.982246 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:58.982247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:58.982249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:58.982306 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.982308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.983208 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.983209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.984879 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:58.984882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:58.985938 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:58.985939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:58.986006 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:58.986010 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:58.986060 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 12:15:58.986110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260925 12:15:58.987432 19705 tablet_service.cc:1467] Processing CreateTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 (DEFAULT_TABLE table=test-table [id=ca4b4a967da64249b55c4cc07aa2052c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:58.987432 19839 tablet_service.cc:1467] Processing CreateTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 (DEFAULT_TABLE table=test-table [id=ca4b4a967da64249b55c4cc07aa2052c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:58.987509 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 (DEFAULT_TABLE table=test-table [id=ca4b4a967da64249b55c4cc07aa2052c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:58.987809 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e1c59080c543a680c93bf1d91c54d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:58.987809 19705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e1c59080c543a680c93bf1d91c54d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:58.987825 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e1c59080c543a680c93bf1d91c54d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:58.990698 20056 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Bootstrap starting.
I20260925 12:15:58.991274 20058 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Bootstrap starting.
I20260925 12:15:58.991340 20056 tablet_bootstrap.cc:654] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:58.991504 20057 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Bootstrap starting.
I20260925 12:15:58.992000 20056 log.cc:824] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:58.992197 20057 tablet_bootstrap.cc:654] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:58.992846 20057 log.cc:824] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:58.993115 20058 tablet_bootstrap.cc:654] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:58.993230 20056 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: No bootstrap required, opened a new log
I20260925 12:15:58.993304 20056 ts_tablet_manager.cc:1397] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 12:15:58.993692 20057 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: No bootstrap required, opened a new log
I20260925 12:15:58.993736 20058 log.cc:824] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:58.993767 20057 ts_tablet_manager.cc:1397] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:15:58.994776 20056 raft_consensus.cc:348] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.994940 20056 raft_consensus.cc:374] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:58.994987 20056 raft_consensus.cc:729] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954fa6bc29204339945cfdecda1df1be, State: Initialized, Role: FOLLOWER
I20260925 12:15:58.995111 20056 consensus_queue.cc:260] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.995261 20057 raft_consensus.cc:348] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.995412 20056 ts_tablet_manager.cc:1428] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:58.995450 20039 heartbeater.cc:503] Master 127.0.98.62:38945 was elected leader, sending a full tablet report...
I20260925 12:15:58.995442 20057 raft_consensus.cc:374] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:58.995492 20057 raft_consensus.cc:729] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1010f133cd4db7a58931ad57461365, State: Initialized, Role: FOLLOWER
I20260925 12:15:58.995625 20057 consensus_queue.cc:260] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.995901 20057 ts_tablet_manager.cc:1428] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 12:15:58.996016 19905 heartbeater.cc:503] Master 127.0.98.62:38945 was elected leader, sending a full tablet report...
I20260925 12:15:58.996050 20058 tablet_bootstrap.cc:492] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: No bootstrap required, opened a new log
I20260925 12:15:58.996102 20058 ts_tablet_manager.cc:1397] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 12:15:58.997777 20058 raft_consensus.cc:348] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.997905 20058 raft_consensus.cc:374] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:58.997925 20058 raft_consensus.cc:729] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e740730a9e5247a1a727dc556406c5a4, State: Initialized, Role: FOLLOWER
I20260925 12:15:58.998060 20058 consensus_queue.cc:260] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:58.998314 20058 ts_tablet_manager.cc:1428] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 12:15:58.998351 19771 heartbeater.cc:503] Master 127.0.98.62:38945 was elected leader, sending a full tablet report...
W20260925 12:15:59.038286 19906 tablet.cc:2369] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 12:15:59.129772 19772 tablet.cc:2369] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:59.137180 20063 raft_consensus.cc:482] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:59.137301 20063 raft_consensus.cc:504] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:59.137758 20063 leader_election.cc:288] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e740730a9e5247a1a727dc556406c5a4 (127.0.98.1:41231), 954fa6bc29204339945cfdecda1df1be (127.0.98.3:38373)
I20260925 12:15:59.140762 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.137950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38373 (local address 127.0.98.2:54297)
0925 12:15:59.138045 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.138060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.138129 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.138292 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.138293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.138295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.138373 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.138376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.139074 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.139074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.139886 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.139889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.140456 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.140458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.140565 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.140569 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.140635 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.140673 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 12:15:59.140785 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.138074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54297 (local address 127.0.98.3:38373)
0925 12:15:59.138127 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.138130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.138142 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:15:59.138203 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.138204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.138215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.138271 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.138454 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.138966 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.139985 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.140385 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.140439 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.140447 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.140641 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.140702 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 12:15:59.141048 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.138095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56681 (local address 127.0.98.1:41231)
0925 12:15:59.138139 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.138141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.138275 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:15:59.138304 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.138305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.138318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.138385 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.138606 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.139101 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.140077 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.140677 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.140739 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.140748 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.140943 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.140991 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:15:59.141103 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e1c59080c543a680c93bf1d91c54d0" candidate_uuid: "be1010f133cd4db7a58931ad57461365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fa6bc29204339945cfdecda1df1be" is_pre_election: true
I20260925 12:15:59.141268 19993 raft_consensus.cc:2454] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1010f133cd4db7a58931ad57461365 in term 0.
I20260925 12:15:59.141301 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.137950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41231 (local address 127.0.98.2:56681)
0925 12:15:59.138155 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.138189 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.138251 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.138396 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.138396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.138398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.138492 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.138493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.139195 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.139196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.139974 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.139977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.140758 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.140759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.140837 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.140842 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.141144 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.141216 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":14,"thread_start_us":112,"threads_started":1}
I20260925 12:15:59.141546 19793 leader_election.cc:302] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 954fa6bc29204339945cfdecda1df1be, be1010f133cd4db7a58931ad57461365; no voters:
I20260925 12:15:59.141603 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e1c59080c543a680c93bf1d91c54d0" candidate_uuid: "be1010f133cd4db7a58931ad57461365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e740730a9e5247a1a727dc556406c5a4" is_pre_election: true
I20260925 12:15:59.141682 20063 raft_consensus.cc:2781] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:59.141728 20063 raft_consensus.cc:482] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:59.141748 20063 raft_consensus.cc:3037] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.141789 19725 raft_consensus.cc:2454] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1010f133cd4db7a58931ad57461365 in term 0.
I20260925 12:15:59.142503 20063 raft_consensus.cc:504] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:59.142643 20063 leader_election.cc:288] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [CANDIDATE]: Term 1 election: Requested vote from peers e740730a9e5247a1a727dc556406c5a4 (127.0.98.1:41231), 954fa6bc29204339945cfdecda1df1be (127.0.98.3:38373)
I20260925 12:15:59.142773 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e1c59080c543a680c93bf1d91c54d0" candidate_uuid: "be1010f133cd4db7a58931ad57461365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fa6bc29204339945cfdecda1df1be"
I20260925 12:15:59.142844 19993 raft_consensus.cc:3037] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.142802 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e1c59080c543a680c93bf1d91c54d0" candidate_uuid: "be1010f133cd4db7a58931ad57461365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e740730a9e5247a1a727dc556406c5a4"
I20260925 12:15:59.142896 19725 raft_consensus.cc:3037] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.143599 19993 raft_consensus.cc:2454] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1010f133cd4db7a58931ad57461365 in term 1.
I20260925 12:15:59.143748 19793 leader_election.cc:302] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 954fa6bc29204339945cfdecda1df1be, be1010f133cd4db7a58931ad57461365; no voters:
I20260925 12:15:59.143800 19725 raft_consensus.cc:2454] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1010f133cd4db7a58931ad57461365 in term 1.
I20260925 12:15:59.143883 20063 raft_consensus.cc:2781] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:59.144052 20063 raft_consensus.cc:686] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 1 LEADER]: Becoming Leader. State: Replica: be1010f133cd4db7a58931ad57461365, State: Running, Role: LEADER
I20260925 12:15:59.144153 20063 consensus_queue.cc:237] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } }
I20260925 12:15:59.145015 19580 catalog_manager.cc:5495] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1010f133cd4db7a58931ad57461365 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "be1010f133cd4db7a58931ad57461365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1010f133cd4db7a58931ad57461365" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20260925 12:15:59.165540 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.163245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.2:34275)
0925 12:15:59.163289 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.163291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.163301 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:15:59.163359 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.163360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.163378 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.163430 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:59.163589 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.164073 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.164816 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.164998 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.165052 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.165225 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:59.165328 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:59.165396 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.165444 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.165460 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:15:59.167356 19725 raft_consensus.cc:1261] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 1 FOLLOWER]: Refusing update from remote peer be1010f133cd4db7a58931ad57461365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:59.167375 19993 raft_consensus.cc:1261] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 1 FOLLOWER]: Refusing update from remote peer be1010f133cd4db7a58931ad57461365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:59.167708 20066 consensus_queue.cc:1035] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e740730a9e5247a1a727dc556406c5a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:59.167835 20063 consensus_queue.cc:1035] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954fa6bc29204339945cfdecda1df1be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:59.170239 20069 mvcc.cc:205] Tried to move back new op lower bound from 7333226738346106880 to 7333226738254409728. Current Snapshot: MvccSnapshot[applied={T|T < 7333226738346106880}]
I20260925 12:15:59.170881 20070 mvcc.cc:205] Tried to move back new op lower bound from 7333226738346106880 to 7333226738254409728. Current Snapshot: MvccSnapshot[applied={T|T < 7333226738346106880}]
I20260925 12:15:59.171249 20071 mvcc.cc:205] Tried to move back new op lower bound from 7333226738346106880 to 7333226738254409728. Current Snapshot: MvccSnapshot[applied={T|T < 7333226738346106880 or (T in {7333226738346106880})}]
I20260925 12:15:59.172354 19579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:59.172468 19579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:59.173260 19579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be4015ba4a44d8eb40e333b85e33657: Sending DeleteTablet for 3 replicas of tablet 67e1c59080c543a680c93bf1d91c54d0
I20260925 12:15:59.173575 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59872
I20260925 12:15:59.173605 19705 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46880
I20260925 12:15:59.173743 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e1c59080c543a680c93bf1d91c54d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51892
I20260925 12:15:59.173843 20080 tablet_replica.cc:330] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: stopping tablet replica
I20260925 12:15:59.173934 20080 raft_consensus.cc:2229] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:59.173947 20081 tablet_replica.cc:330] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: stopping tablet replica
I20260925 12:15:59.174034 20081 raft_consensus.cc:2229] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:59.174046 20080 raft_consensus.cc:2258] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:59.174130 20081 raft_consensus.cc:2258] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:59.174420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19642
I20260925 12:15:59.174495 20080 ts_tablet_manager.cc:1905] T 67e1c59080c543a680c93bf1d91c54d0 P e740730a9e5247a1a727dc556406c5a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:59.174553 20081 ts_tablet_manager.cc:1905] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:59.177139 20081 ts_tablet_manager.cc:1918] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:59.177218 20081 log.cc:1196] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/67e1c59080c543a680c93bf1d91c54d0
I20260925 12:15:59.177489 20081 ts_tablet_manager.cc:1939] T 67e1c59080c543a680c93bf1d91c54d0 P 954fa6bc29204339945cfdecda1df1be: Deleting consensus metadata
I20260925 12:15:59.177733 20079 tablet_replica.cc:330] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: stopping tablet replica
I20260925 12:15:59.177860 19565 catalog_manager.cc:4841] TS 954fa6bc29204339945cfdecda1df1be (127.0.98.3:38373): tablet 67e1c59080c543a680c93bf1d91c54d0 (table test-table [id=ca4b4a967da64249b55c4cc07aa2052c]) successfully deleted
I20260925 12:15:59.177901 20079 raft_consensus.cc:2229] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:15:59.178025 20079 raft_consensus.cc:2258] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:59.178406 20079 ts_tablet_manager.cc:1905] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:59.179978 20079 ts_tablet_manager.cc:1918] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:59.180037 20079 log.cc:1196] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/67e1c59080c543a680c93bf1d91c54d0
I20260925 12:15:59.180267 20079 ts_tablet_manager.cc:1939] T 67e1c59080c543a680c93bf1d91c54d0 P be1010f133cd4db7a58931ad57461365: Deleting consensus metadata
I20260925 12:15:59.180819 19567 catalog_manager.cc:4841] TS be1010f133cd4db7a58931ad57461365 (127.0.98.2:34275): tablet 67e1c59080c543a680c93bf1d91c54d0 (table test-table [id=ca4b4a967da64249b55c4cc07aa2052c]) successfully deleted
W20260925 12:15:59.181071 19567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41231 (error 108)
W20260925 12:15:59.181593 20053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:15:59.181448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41231 (local address 127.0.0.1:46884)
0925 12:15:59.181510 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.181557 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41231: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 12:15:59.181716 19567 catalog_manager.cc:4584] TS e740730a9e5247a1a727dc556406c5a4 (127.0.98.1:41231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67e1c59080c543a680c93bf1d91c54d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41231: connect: Connection reset by peer (error 104)
I20260925 12:15:59.181919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19776
I20260925 12:15:59.186949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19910
I20260925 12:15:59.191291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 19548
2026-09-25T12:15:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T12:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:15:59Z Disabled control of system clock
I20260925 12:15:59.219501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35923
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:59.300182 20086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:59.300414 20086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:59.300436 20086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:59.301651 20086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:15:59.301702 20086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:59.301715 20086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:59.301728 20086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:15:59.301746 20086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:15:59.303035 20086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:59.303229 20086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:59.303534 20086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:59.308342 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.308482 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:59.308475 20086 server_base.cc:1034] running on GCE node
W20260925 12:15:59.308426 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:59.308890 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:59.309171 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:59.310334 20086 hybrid_clock.cc:648] HybridClock initialized: now 1790338559310331 us; error 36 us; skew 500 ppm
I20260925 12:15:59.311605 20086 webserver.cc:460] Webserver started at http://127.0.98.62:42195/ using document root <none> and password file <none>
I20260925 12:15:59.311841 20086 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:59.311908 20086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:59.312057 20086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:59.313199 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "f7ec48f3deb5460fb43801bcf03d6449"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "0a0f9d6942801ec233c8349f09512f59"
server_key_iv: "5a86313beb0e1dba111d1b05a2ddff66"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.313582 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7ec48f3deb5460fb43801bcf03d6449"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "0a0f9d6942801ec233c8349f09512f59"
server_key_iv: "5a86313beb0e1dba111d1b05a2ddff66"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.314942 20086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 12:15:59.315649 20100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.315824 20086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:59.315923 20086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "f7ec48f3deb5460fb43801bcf03d6449"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "0a0f9d6942801ec233c8349f09512f59"
server_key_iv: "5a86313beb0e1dba111d1b05a2ddff66"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.316004 20086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:59.326232 20086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:59.326521 20086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:59.326683 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:59.330714 20086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35923
I20260925 12:15:59.330746 20152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35923 every 8 connection(s)
I20260925 12:15:59.331105 20086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:15:59.332028 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:59.334012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20086
I20260925 12:15:59.334121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:15:59.334349 392 external_mini_cluster.cc:1428] Setting key 2025b74368aa34e819e21eb5237b0573
I20260925 12:15:59.334569 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: Bootstrap starting.
I20260925 12:15:59.335238 20153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:59.335558 20153 log.cc:824] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:59.336313 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: No bootstrap required, opened a new log
I20260925 12:15:59.338681 20153 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } }
I20260925 12:15:59.338865 20153 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:59.338908 20153 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec48f3deb5460fb43801bcf03d6449, State: Initialized, Role: FOLLOWER
I20260925 12:15:59.339035 20153 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } }
I20260925 12:15:59.339138 20153 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:15:59.339172 20153 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:15:59.339218 20153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.339862 20153 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } }
I20260925 12:15:59.339993 20153 leader_election.cc:302] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ec48f3deb5460fb43801bcf03d6449; no voters:
I20260925 12:15:59.340103 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.334877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.0.98.62:35923)
0925 12:15:59.335102 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.335117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.335140 (+ 23us) server_negotiation.cc:407] Connection header received
0925 12:15:59.335598 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.335605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.335693 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.335866 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:59.336093 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.337058 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.338183 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.338561 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.338642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.339397 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:59.339400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:59.339400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:59.339556 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:59.339587 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:59.339608 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:59.339658 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.339782 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.339864 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 12:15:59.340173 20153 leader_election.cc:288] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:15:59.340206 20158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:59.340349 20158 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [term 1 LEADER]: Becoming Leader. State: Replica: f7ec48f3deb5460fb43801bcf03d6449, State: Running, Role: LEADER
I20260925 12:15:59.340417 20153 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:15:59.340438 20158 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } }
I20260925 12:15:59.340795 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ec48f3deb5460fb43801bcf03d6449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } } }
I20260925 12:15:59.340869 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:59.341004 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ec48f3deb5460fb43801bcf03d6449. Latest consensus state: current_term: 1 leader_uuid: "f7ec48f3deb5460fb43801bcf03d6449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ec48f3deb5460fb43801bcf03d6449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35923 } } }
I20260925 12:15:59.341068 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449 [sys.catalog]: This master's current role is: LEADER
I20260925 12:15:59.341090 20163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:15:59.341578 20163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:15:59.343178 20163 catalog_manager.cc:1347] Generated new cluster ID: 4ae8596f88cc4916921abbba83bbf9ce
I20260925 12:15:59.343232 20163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:15:59.357224 20163 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:15:59.357815 20163 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:15:59.361913 20163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: Generated new TSK 0
I20260925 12:15:59.362099 20163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:15:59.368309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:59.449229 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:59.449424 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:59.449445 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:59.450583 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:59.450624 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:59.450649 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 12:15:59.451974 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:59.452208 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:59.452562 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:59.458313 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.458290 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.458276 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:59.458495 20177 server_base.cc:1034] running on GCE node
I20260925 12:15:59.458746 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:59.459025 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:59.460193 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790338559460175 us; error 32 us; skew 500 ppm
I20260925 12:15:59.461633 20177 webserver.cc:460] Webserver started at http://127.0.98.1:41699/ using document root <none> and password file <none>
I20260925 12:15:59.461901 20177 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:59.461972 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:59.462122 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:59.463524 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3731d64aa374f3e97794e56a18039da"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "440e989593fa6653ebe4865d5245ce51"
server_key_iv: "f034a0ed75766b98ff7407d8664319a2"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.464010 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3731d64aa374f3e97794e56a18039da"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "440e989593fa6653ebe4865d5245ce51"
server_key_iv: "f034a0ed75766b98ff7407d8664319a2"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.465732 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260925 12:15:59.466686 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.466892 20177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:59.467029 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal
uuid: "d3731d64aa374f3e97794e56a18039da"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "440e989593fa6653ebe4865d5245ce51"
server_key_iv: "f034a0ed75766b98ff7407d8664319a2"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.467116 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:59.486538 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:59.486816 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:59.486924 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:59.487113 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:59.487488 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:59.487525 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.487588 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:59.487636 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.493484 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46851
I20260925 12:15:59.493578 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46851 every 8 connection(s)
I20260925 12:15:59.493902 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/data/info.pb
I20260925 12:15:59.499070 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.494689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58405 (local address 127.0.98.62:35923)
0925 12:15:59.494777 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.494780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.495031 (+ 251us) server_negotiation.cc:407] Connection header received
0925 12:15:59.495508 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.495509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.495519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.495568 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:59.496226 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.496798 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.497752 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.497950 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.498000 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.498867 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:59.498868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:59.498868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:59.498872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:59.498882 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:59.498886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:59.498923 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.498991 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.499010 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 12:15:59.499305 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.494643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35923 (local address 127.0.98.1:58405)
0925 12:15:59.494833 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.494920 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.495418 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.495630 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.495633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.495641 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:59.496151 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.496155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.496899 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.496900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.497677 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.497679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.498028 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.498029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.498139 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.498727 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:59.498728 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:59.498757 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:59.498780 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:59.498925 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:59.498927 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:59.498929 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.498977 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.499056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 12:15:59.499804 20305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35923
I20260925 12:15:59.499914 20305 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:59.500056 20305 heartbeater.cc:511] Master 127.0.98.62:35923 requested a full tablet report, sending...
I20260925 12:15:59.500566 20117 ts_manager.cc:194] Registered new tserver with Master: d3731d64aa374f3e97794e56a18039da (127.0.98.1:46851)
I20260925 12:15:59.501179 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58405
I20260925 12:15:59.503639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20177
I20260925 12:15:59.503721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-0/wal/instance
I20260925 12:15:59.503952 392 external_mini_cluster.cc:1428] Setting key 6e24b2bfb9d04c79c1ceac77786fe47b
I20260925 12:15:59.504930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:59.586043 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:59.586294 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:59.586333 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:59.587486 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:59.587555 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:59.587600 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 12:15:59.589036 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:59.589299 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:59.589609 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:59.595005 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.595001 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.595106 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:59.595155 20308 server_base.cc:1034] running on GCE node
I20260925 12:15:59.595528 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:59.595799 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:59.596979 20308 hybrid_clock.cc:648] HybridClock initialized: now 1790338559596968 us; error 49 us; skew 500 ppm
I20260925 12:15:59.598318 20308 webserver.cc:460] Webserver started at http://127.0.98.2:43933/ using document root <none> and password file <none>
I20260925 12:15:59.598595 20308 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:59.598670 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:59.598817 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:59.599961 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data/instance:
uuid: "4b071a84703342b290866dcbf9040a87"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "2fc8ba7699ebdc7ac3beb985f9e82dc6"
server_key_iv: "8350f35a952e2c5ed436cdac9c986cf5"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.600335 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4b071a84703342b290866dcbf9040a87"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "2fc8ba7699ebdc7ac3beb985f9e82dc6"
server_key_iv: "8350f35a952e2c5ed436cdac9c986cf5"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.601704 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:15:59.602474 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.602646 20308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:15:59.602746 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal
uuid: "4b071a84703342b290866dcbf9040a87"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "2fc8ba7699ebdc7ac3beb985f9e82dc6"
server_key_iv: "8350f35a952e2c5ed436cdac9c986cf5"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.602828 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:59.614988 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:59.615293 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:59.615444 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:59.615660 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:59.615988 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:59.616042 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.616084 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:59.616112 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.622323 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46819
I20260925 12:15:59.622373 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46819 every 8 connection(s)
I20260925 12:15:59.622742 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/data/info.pb
I20260925 12:15:59.627789 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.623480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42427 (local address 127.0.98.62:35923)
0925 12:15:59.623565 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.623568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.623778 (+ 210us) server_negotiation.cc:407] Connection header received
0925 12:15:59.624268 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.624269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.624279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.624333 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:59.625059 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.625547 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.626541 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.626724 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.626775 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.627574 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:59.627575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:59.627575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:59.627580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:59.627593 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:59.627597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:59.627634 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.627702 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.627724 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 12:15:59.628036 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.623445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35923 (local address 127.0.98.2:42427)
0925 12:15:59.623659 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.623693 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.624181 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.624390 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.624394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.624404 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:59.624981 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.624987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.625626 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.625627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.626462 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.626464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.626800 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.626801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.626948 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.627440 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:59.627441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:59.627470 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:59.627493 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:59.627641 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:59.627643 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:59.627646 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.627693 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.627769 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260925 12:15:59.628698 20436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35923
I20260925 12:15:59.628790 20436 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:59.628988 20436 heartbeater.cc:511] Master 127.0.98.62:35923 requested a full tablet report, sending...
I20260925 12:15:59.629349 20117 ts_manager.cc:194] Registered new tserver with Master: 4b071a84703342b290866dcbf9040a87 (127.0.98.2:46819)
I20260925 12:15:59.629786 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42427
I20260925 12:15:59.629868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20308
I20260925 12:15:59.629984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal/instance
I20260925 12:15:59.630191 392 external_mini_cluster.cc:1428] Setting key 05e2905cb3c1f650e99493afd3c207ec
I20260925 12:15:59.631212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 12:15:59.712272 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:15:59.712473 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:15:59.712510 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:15:59.713663 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:15:59.713704 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:15:59.713729 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 12:15:59.715013 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:15:59.715205 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:15:59.715445 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:15:59.720705 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.720698 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:15:59.720693 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:15:59.720769 20439 server_base.cc:1034] running on GCE node
I20260925 12:15:59.721171 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:15:59.721379 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:15:59.722513 20439 hybrid_clock.cc:648] HybridClock initialized: now 1790338559722486 us; error 42 us; skew 500 ppm
I20260925 12:15:59.723753 20439 webserver.cc:460] Webserver started at http://127.0.98.3:43331/ using document root <none> and password file <none>
I20260925 12:15:59.723981 20439 fs_manager.cc:359] Metadata directory not provided
I20260925 12:15:59.724028 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:15:59.724123 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:15:59.725234 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data/instance:
uuid: "40bd3e1506ec49adb1fbc2d7c4718929"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "cc0b73c695dc3806641fa5b1ad2fdcd9"
server_key_iv: "762448c7d6883dbe6178f398fe960f85"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.725536 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40bd3e1506ec49adb1fbc2d7c4718929"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "cc0b73c695dc3806641fa5b1ad2fdcd9"
server_key_iv: "762448c7d6883dbe6178f398fe960f85"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.726752 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:15:59.727476 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.727728 20439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:15:59.727840 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal
uuid: "40bd3e1506ec49adb1fbc2d7c4718929"
format_stamp: "Formatted at 2026-09-25 12:15:59 on dist-test-slave-2hqq"
server_key: "cc0b73c695dc3806641fa5b1ad2fdcd9"
server_key_iv: "762448c7d6883dbe6178f398fe960f85"
server_key_version: "encryptionkey@0"
I20260925 12:15:59.727918 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:15:59.738411 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:15:59.738720 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:15:59.738879 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:15:59.739092 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 12:15:59.739411 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 12:15:59.739464 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.739506 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 12:15:59.739533 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 12:15:59.745417 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40285
I20260925 12:15:59.745509 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40285 every 8 connection(s)
I20260925 12:15:59.745862 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/data/info.pb
I20260925 12:15:59.746204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20439
I20260925 12:15:59.746315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal/instance
I20260925 12:15:59.746516 392 external_mini_cluster.cc:1428] Setting key e62159ecbff6122c4e358f9b8705f6f3
I20260925 12:15:59.751456 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.746749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41751 (local address 127.0.98.62:35923)
0925 12:15:59.746802 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.746804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.747061 (+ 257us) server_negotiation.cc:407] Connection header received
0925 12:15:59.747661 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.747661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.747672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.747720 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:59.748538 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.749192 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.750139 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.750327 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.750386 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.751231 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:59.751232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:59.751232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:59.751236 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:59.751247 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:59.751250 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:59.751291 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.751382 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.751399 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:15:59.751713 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.746688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35923 (local address 127.0.98.3:41751)
0925 12:15:59.746916 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.746967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.747580 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.747773 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.747777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.747785 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:15:59.748410 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.748416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.749287 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.749288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.750065 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.750067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.750426 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.750427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.750556 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.751100 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 12:15:59.751100 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:15:59.751129 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 12:15:59.751153 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:15:59.751307 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:15:59.751311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:15:59.751314 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.751373 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.751453 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260925 12:15:59.752208 20567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35923
I20260925 12:15:59.752297 20567 heartbeater.cc:464] Registering TS with master...
I20260925 12:15:59.752431 20567 heartbeater.cc:511] Master 127.0.98.62:35923 requested a full tablet report, sending...
I20260925 12:15:59.752862 20117 ts_manager.cc:194] Registered new tserver with Master: 40bd3e1506ec49adb1fbc2d7c4718929 (127.0.98.3:40285)
I20260925 12:15:59.753237 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41751
I20260925 12:15:59.758469 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 12:15:59.762586 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.759941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.62:35923)
0925 12:15:59.760020 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.760022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.760030 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:15:59.760081 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.760081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.760089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.760135 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:15:59.760264 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.760782 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.761537 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.761709 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.761758 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.762342 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:15:59.762343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:15:59.762344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:15:59.762348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:15:59.762360 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:15:59.762363 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:15:59.762404 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.762463 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.762483 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 12:15:59.764176 20117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 12:15:59.765892 20117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 12:15:59.772264 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46819 (local address 127.0.0.1:60324)
0925 12:15:59.768569 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.768580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.768634 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.768886 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.768886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.768890 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.768938 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.768940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.769806 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.769807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.771034 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.771036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.771981 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.771982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.772055 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.772071 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.772135 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.772187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260925 12:15:59.772408 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46851 (local address 127.0.0.1:46084)
0925 12:15:59.768119 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.768155 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.768240 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.768831 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.768834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.768836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.768900 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.768902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.770198 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.770199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.771208 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.771209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.772147 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.772148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.772211 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.772216 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.772271 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.772312 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260925 12:15:59.772619 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.98.2:46819)
0925 12:15:59.768539 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.768543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.768682 (+ 139us) server_negotiation.cc:407] Connection header received
0925 12:15:59.768725 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.768727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.768742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.768817 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.769028 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.769709 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.771114 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.771879 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.771970 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.772072 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.772375 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.772417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260925 12:15:59.772727 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.98.1:46851)
0925 12:15:59.768551 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.768556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.768567 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:59.768610 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.768612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.768630 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.768732 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.768970 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.770107 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.771281 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.772073 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.772135 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.772210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.772284 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.772595 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":86,"threads_started":1}
I20260925 12:15:59.773288 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.0.1:50816)
0925 12:15:59.768328 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.768338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.768388 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.768885 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.768886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.768888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.768937 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.768939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.771273 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.771273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.772042 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.772044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.773057 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.773057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.773117 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.773120 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.773176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.773215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":10,"thread_start_us":85,"threads_started":1}
I20260925 12:15:59.773322 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.768332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.0.98.3:40285)
0925 12:15:59.768597 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.768600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.768609 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:15:59.768648 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.768650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.768666 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.768817 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.769008 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.770096 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.772135 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.772866 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.772930 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.773023 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.773205 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.773245 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260925 12:15:59.773270 20370 tablet_service.cc:1467] Processing CreateTablet for tablet ca6605079b72463cbbbcd2a2b03019fc (DEFAULT_TABLE table=test-table [id=49aec906f84141d2bab0fd1999bea377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:59.773398 20239 tablet_service.cc:1467] Processing CreateTablet for tablet ca6605079b72463cbbbcd2a2b03019fc (DEFAULT_TABLE table=test-table [id=49aec906f84141d2bab0fd1999bea377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:59.773684 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6605079b72463cbbbcd2a2b03019fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:59.773725 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6605079b72463cbbbcd2a2b03019fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:59.774041 20501 tablet_service.cc:1467] Processing CreateTablet for tablet ca6605079b72463cbbbcd2a2b03019fc (DEFAULT_TABLE table=test-table [id=49aec906f84141d2bab0fd1999bea377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 12:15:59.774325 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6605079b72463cbbbcd2a2b03019fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:15:59.776081 20581 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Bootstrap starting.
I20260925 12:15:59.776695 20581 tablet_bootstrap.cc:654] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:59.776981 20582 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: Bootstrap starting.
I20260925 12:15:59.777006 20581 log.cc:824] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:59.777563 20584 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Bootstrap starting.
I20260925 12:15:59.777614 20582 tablet_bootstrap.cc:654] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:59.777921 20582 log.cc:824] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:59.778187 20584 tablet_bootstrap.cc:654] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Neither blocks nor log segments found. Creating new log.
I20260925 12:15:59.778213 20581 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: No bootstrap required, opened a new log
I20260925 12:15:59.778281 20581 ts_tablet_manager.cc:1397] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:59.778403 20584 log.cc:824] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Log is configured to *not* fsync() on all Append() calls
I20260925 12:15:59.778853 20582 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: No bootstrap required, opened a new log
I20260925 12:15:59.778923 20582 ts_tablet_manager.cc:1397] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:59.779125 20584 tablet_bootstrap.cc:492] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: No bootstrap required, opened a new log
I20260925 12:15:59.779191 20584 ts_tablet_manager.cc:1397] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 12:15:59.780691 20581 raft_consensus.cc:348] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.780844 20581 raft_consensus.cc:374] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:59.780902 20581 raft_consensus.cc:729] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b071a84703342b290866dcbf9040a87, State: Initialized, Role: FOLLOWER
I20260925 12:15:59.781042 20581 consensus_queue.cc:260] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.781309 20581 ts_tablet_manager.cc:1428] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:59.781457 20436 heartbeater.cc:503] Master 127.0.98.62:35923 was elected leader, sending a full tablet report...
I20260925 12:15:59.781570 20582 raft_consensus.cc:348] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.781721 20582 raft_consensus.cc:374] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:59.781762 20582 raft_consensus.cc:729] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3731d64aa374f3e97794e56a18039da, State: Initialized, Role: FOLLOWER
I20260925 12:15:59.781872 20582 consensus_queue.cc:260] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.782112 20582 ts_tablet_manager.cc:1428] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 12:15:59.782256 20305 heartbeater.cc:503] Master 127.0.98.62:35923 was elected leader, sending a full tablet report...
I20260925 12:15:59.782220 20584 raft_consensus.cc:348] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.782364 20584 raft_consensus.cc:374] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:15:59.782406 20584 raft_consensus.cc:729] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bd3e1506ec49adb1fbc2d7c4718929, State: Initialized, Role: FOLLOWER
I20260925 12:15:59.782517 20584 consensus_queue.cc:260] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.782773 20584 ts_tablet_manager.cc:1428] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 12:15:59.782928 20567 heartbeater.cc:503] Master 127.0.98.62:35923 was elected leader, sending a full tablet report...
I20260925 12:15:59.871248 20587 raft_consensus.cc:482] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 12:15:59.871372 20587 raft_consensus.cc:504] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.871835 20587 leader_election.cc:288] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3731d64aa374f3e97794e56a18039da (127.0.98.1:46851), 40bd3e1506ec49adb1fbc2d7c4718929 (127.0.98.3:40285)
W20260925 12:15:59.873255 20437 tablet.cc:2369] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 12:15:59.875444 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.872041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46851 (local address 127.0.98.2:49447)
0925 12:15:59.872105 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.872116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.872163 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.872332 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.872334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.872335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.872408 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.872415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.873417 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.873418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.874608 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.874610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.875188 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.875189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.875270 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.875273 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.875314 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.875346 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":3}
I20260925 12:15:59.875456 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.872130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49447 (local address 127.0.98.1:46851)
0925 12:15:59.872178 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.872180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.872191 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:15:59.872227 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.872228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.872238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.872309 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.872530 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.873306 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.874713 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.875079 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.875169 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.875180 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.875338 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.875385 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:15:59.875509 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.872164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57735 (local address 127.0.98.3:40285)
0925 12:15:59.872213 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.872215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.872370 (+ 155us) server_negotiation.cc:407] Connection header received
0925 12:15:59.872406 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.872407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.872415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.872454 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 12:15:59.872634 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.873352 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.874722 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.875152 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.875221 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.875231 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.875393 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.875434 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 12:15:59.875540 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.872065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.98.2:57735)
0925 12:15:59.872260 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.872267 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 12:15:59.872332 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:15:59.872505 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:15:59.872507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:15:59.872508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 12:15:59.872568 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.872569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.873433 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.873434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.874630 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:15:59.874633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.875238 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.875239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:15:59.875295 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.875318 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 12:15:59.875375 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 12:15:59.875409 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":113,"thread_start_us":96,"threads_started":1}
I20260925 12:15:59.875874 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6605079b72463cbbbcd2a2b03019fc" candidate_uuid: "4b071a84703342b290866dcbf9040a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" is_pre_election: true
I20260925 12:15:59.875994 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6605079b72463cbbbcd2a2b03019fc" candidate_uuid: "4b071a84703342b290866dcbf9040a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3731d64aa374f3e97794e56a18039da" is_pre_election: true
I20260925 12:15:59.876045 20521 raft_consensus.cc:2454] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b071a84703342b290866dcbf9040a87 in term 0.
I20260925 12:15:59.876135 20259 raft_consensus.cc:2454] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b071a84703342b290866dcbf9040a87 in term 0.
I20260925 12:15:59.876293 20325 leader_election.cc:302] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bd3e1506ec49adb1fbc2d7c4718929, 4b071a84703342b290866dcbf9040a87; no voters:
I20260925 12:15:59.876430 20587 raft_consensus.cc:2781] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 12:15:59.876495 20587 raft_consensus.cc:482] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 12:15:59.876560 20587 raft_consensus.cc:3037] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.877252 20587 raft_consensus.cc:504] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.877403 20587 leader_election.cc:288] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [CANDIDATE]: Term 1 election: Requested vote from peers d3731d64aa374f3e97794e56a18039da (127.0.98.1:46851), 40bd3e1506ec49adb1fbc2d7c4718929 (127.0.98.3:40285)
I20260925 12:15:59.877559 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6605079b72463cbbbcd2a2b03019fc" candidate_uuid: "4b071a84703342b290866dcbf9040a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd3e1506ec49adb1fbc2d7c4718929"
I20260925 12:15:59.877559 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6605079b72463cbbbcd2a2b03019fc" candidate_uuid: "4b071a84703342b290866dcbf9040a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3731d64aa374f3e97794e56a18039da"
I20260925 12:15:59.877636 20259 raft_consensus.cc:3037] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.877660 20521 raft_consensus.cc:3037] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:15:59.878386 20259 raft_consensus.cc:2454] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b071a84703342b290866dcbf9040a87 in term 1.
I20260925 12:15:59.878404 20521 raft_consensus.cc:2454] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b071a84703342b290866dcbf9040a87 in term 1.
I20260925 12:15:59.878573 20324 leader_election.cc:302] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b071a84703342b290866dcbf9040a87, d3731d64aa374f3e97794e56a18039da; no voters:
I20260925 12:15:59.878664 20587 raft_consensus.cc:2781] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:15:59.878796 20587 raft_consensus.cc:686] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 1 LEADER]: Becoming Leader. State: Replica: 4b071a84703342b290866dcbf9040a87, State: Running, Role: LEADER
I20260925 12:15:59.878891 20587 consensus_queue.cc:237] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } }
I20260925 12:15:59.879771 20116 catalog_manager.cc:5495] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b071a84703342b290866dcbf9040a87 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4b071a84703342b290866dcbf9040a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b071a84703342b290866dcbf9040a87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46819 } health_report { overall_health: HEALTHY } } }
I20260925 12:15:59.890769 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.888602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.0.98.2:46819)
0925 12:15:59.888644 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:15:59.888646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:15:59.888682 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:15:59.888715 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:15:59.888716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:15:59.888732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:15:59.888784 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:15:59.888897 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.889352 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.890070 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:15:59.890244 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:15:59.890294 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:15:59.890480 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:15:59.890582 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:15:59.890642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:15:59.890677 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 12:15:59.890691 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:15:59.892622 20259 raft_consensus.cc:1261] T ca6605079b72463cbbbcd2a2b03019fc P d3731d64aa374f3e97794e56a18039da [term 1 FOLLOWER]: Refusing update from remote peer 4b071a84703342b290866dcbf9040a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:59.892622 20521 raft_consensus.cc:1261] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 1 FOLLOWER]: Refusing update from remote peer 4b071a84703342b290866dcbf9040a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:15:59.892976 20587 consensus_queue.cc:1035] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3731d64aa374f3e97794e56a18039da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:59.893162 20591 consensus_queue.cc:1035] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bd3e1506ec49adb1fbc2d7c4718929" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:15:59.895328 20594 mvcc.cc:205] Tried to move back new op lower bound from 7333226741316444160 to 7333226741264056320. Current Snapshot: MvccSnapshot[applied={T|T < 7333226741316444160}]
I20260925 12:15:59.896412 20595 mvcc.cc:205] Tried to move back new op lower bound from 7333226741316444160 to 7333226741264056320. Current Snapshot: MvccSnapshot[applied={T|T < 7333226741316444160}]
I20260925 12:15:59.897502 20117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:59.897603 20117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 12:15:59.898325 20117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7ec48f3deb5460fb43801bcf03d6449: Sending DeleteTablet for 3 replicas of tablet ca6605079b72463cbbbcd2a2b03019fc
I20260925 12:15:59.898639 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6605079b72463cbbbcd2a2b03019fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='slave'} at 127.0.0.1:46084
I20260925 12:15:59.898679 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6605079b72463cbbbcd2a2b03019fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='slave'} at 127.0.0.1:50816
I20260925 12:15:59.898909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20177
I20260925 12:15:59.898933 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6605079b72463cbbbcd2a2b03019fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 12:15:59 UTC) from {username='slave'} at 127.0.0.1:60324
I20260925 12:15:59.899124 20605 tablet_replica.cc:330] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: stopping tablet replica
I20260925 12:15:59.899212 20605 raft_consensus.cc:2229] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 12:15:59.899303 20605 raft_consensus.cc:2258] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:59.899466 20606 tablet_replica.cc:330] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: stopping tablet replica
I20260925 12:15:59.899560 20606 raft_consensus.cc:2229] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 1 LEADER]: Raft consensus shutting down.
I20260925 12:15:59.899646 20606 raft_consensus.cc:2258] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 12:15:59.900118 20606 ts_tablet_manager.cc:1905] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:59.900118 20605 ts_tablet_manager.cc:1905] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 12:15:59.902084 20606 ts_tablet_manager.cc:1918] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:59.902213 20606 log.cc:1196] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-1/wal/wals/ca6605079b72463cbbbcd2a2b03019fc
I20260925 12:15:59.902271 20605 ts_tablet_manager.cc:1918] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 12:15:59.902351 20605 log.cc:1196] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Deleting WAL directory at /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790338290997044-392-0/minicluster-data/ts-2/wal/wals/ca6605079b72463cbbbcd2a2b03019fc
I20260925 12:15:59.902642 20606 ts_tablet_manager.cc:1939] T ca6605079b72463cbbbcd2a2b03019fc P 4b071a84703342b290866dcbf9040a87: Deleting consensus metadata
I20260925 12:15:59.902643 20605 ts_tablet_manager.cc:1939] T ca6605079b72463cbbbcd2a2b03019fc P 40bd3e1506ec49adb1fbc2d7c4718929: Deleting consensus metadata
I20260925 12:15:59.903041 20103 catalog_manager.cc:4841] TS 40bd3e1506ec49adb1fbc2d7c4718929 (127.0.98.3:40285): tablet ca6605079b72463cbbbcd2a2b03019fc (table test-table [id=49aec906f84141d2bab0fd1999bea377]) successfully deleted
I20260925 12:15:59.903146 20103 catalog_manager.cc:4841] TS 4b071a84703342b290866dcbf9040a87 (127.0.98.2:46819): tablet ca6605079b72463cbbbcd2a2b03019fc (table test-table [id=49aec906f84141d2bab0fd1999bea377]) successfully deleted
W20260925 12:15:59.906322 20102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46851 (error 108)
I20260925 12:15:59.906605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20308
I20260925 12:15:59.906912 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:15:59.906778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46851 (local address 127.0.0.1:46088)
0925 12:15:59.906830 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:15:59.906872 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 12:15:59.907960 20102 catalog_manager.cc:4584] TS d3731d64aa374f3e97794e56a18039da (127.0.98.1:46851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca6605079b72463cbbbcd2a2b03019fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46851: connect: Connection refused (error 111)
I20260925 12:15:59.910683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20439
I20260925 12:15:59.914933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20086
2026-09-25T12:15:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (721 ms)
[----------] 6 tests from EncryptionPolicyTest (8608 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:15:59 dist-test-slave-2hqq krb5kdc[20610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:15:59 dist-test-slave-2hqq krb5kdc[20610](info): set up 2 sockets
Sep 25 12:15:59 dist-test-slave-2hqq krb5kdc[20610](info): commencing operation
krb5kdc: starting...
W20260925 12:16:00.968652 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:01.091714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:01.174612 20623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:01.174873 20623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:01.174912 20623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:01.176157 20623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:01.176224 20623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:01.176251 20623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:01.176280 20623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:01.176307 20623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:01.177837 20623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:01.178121 20623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:01.178355 20623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:01.183300 20629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:01.183274 20631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:01.183274 20628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:01.183533 20623 server_base.cc:1034] running on GCE node
I20260925 12:16:01.183830 20623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:01.184067 20623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:01.185202 20623 hybrid_clock.cc:648] HybridClock initialized: now 1790338561185188 us; error 28 us; skew 500 ppm
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:01.194514 20623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:01.195010 20623 webserver.cc:460] Webserver started at http://127.0.0.1:37711/ using document root <none> and password file <none>
I20260925 12:16:01.195209 20623 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:01.195256 20623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:01.195348 20623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:01.196456 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "9aae4d6a6979490b9135879183916b6f"
format_stamp: "Formatted at 2026-09-25 12:16:01 on dist-test-slave-2hqq"
server_key: "824ef17b8a6f43e9d3fed0321be5e013"
server_key_iv: "627cce784d28c19a8e7be9fe87566fc3"
server_key_version: "encryptionkey@0"
I20260925 12:16:01.196870 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "9aae4d6a6979490b9135879183916b6f"
format_stamp: "Formatted at 2026-09-25 12:16:01 on dist-test-slave-2hqq"
server_key: "824ef17b8a6f43e9d3fed0321be5e013"
server_key_iv: "627cce784d28c19a8e7be9fe87566fc3"
server_key_version: "encryptionkey@0"
I20260925 12:16:01.198251 20623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:01.199242 20638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:01.199447 20623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:01.199544 20623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "9aae4d6a6979490b9135879183916b6f"
format_stamp: "Formatted at 2026-09-25 12:16:01 on dist-test-slave-2hqq"
server_key: "824ef17b8a6f43e9d3fed0321be5e013"
server_key_iv: "627cce784d28c19a8e7be9fe87566fc3"
server_key_version: "encryptionkey@0"
I20260925 12:16:01.199620 20623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:01.225486 20623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:01.227931 20623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:01.228088 20623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:01.232366 20623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38287
I20260925 12:16:01.232430 20690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38287 every 8 connection(s)
I20260925 12:16:01.232777 20623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:01.233729 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:01.236263 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f: Bootstrap starting.
I20260925 12:16:01.236984 20691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:01.237008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20623
I20260925 12:16:01.237124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:01.237325 392 external_mini_cluster.cc:1428] Setting key a864db51a04569c3f9d4fa1831cfca39
I20260925 12:16:01.237376 20691 log.cc:824] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:01.238260 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f: No bootstrap required, opened a new log
I20260925 12:16:01.239851 20691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260925 12:16:01.240008 20691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:01.240052 20691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aae4d6a6979490b9135879183916b6f, State: Initialized, Role: FOLLOWER
I20260925 12:16:01.240218 20691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260925 12:16:01.240314 20691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:01.240357 20691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:01.240404 20691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:01.240978 20691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260925 12:16:01.241096 20691 leader_election.cc:302] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aae4d6a6979490b9135879183916b6f; no voters:
I20260925 12:16:01.241288 20691 leader_election.cc:288] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:01.241362 20696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:01.241535 20691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:01.241555 20696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [term 1 LEADER]: Becoming Leader. State: Replica: 9aae4d6a6979490b9135879183916b6f, State: Running, Role: LEADER
I20260925 12:16:01.241753 20696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } }
I20260925 12:16:01.242180 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aae4d6a6979490b9135879183916b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20260925 12:16:01.242260 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:01.242425 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aae4d6a6979490b9135879183916b6f. Latest consensus state: current_term: 1 leader_uuid: "9aae4d6a6979490b9135879183916b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aae4d6a6979490b9135879183916b6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38287 } } }
I20260925 12:16:01.242484 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:01.242772 20705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:01.243359 20705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:01.245128 20705 catalog_manager.cc:1347] Generated new cluster ID: bbbb84d90b794052b9c91abe5e39b9b6
I20260925 12:16:01.245182 20705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:16:01.250991 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:01.237919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52429 (local address 127.0.0.1:38287)
0925 12:16:01.238086 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:01.238097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:01.238133 (+ 36us) server_negotiation.cc:407] Connection header received
0925 12:16:01.238679 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:01.238686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:01.238738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:01.238902 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:01.239616 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:01.241059 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:01.242252 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:01.243117 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:01.243266 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:01.246191 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:01.246195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:01.246195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:01.246206 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:01.248467 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:01.249039 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:01.249042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:01.249043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:01.249176 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:01.249454 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:01.249455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:01.249455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:01.249793 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:01.249974 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:01.250481 (+ 507us) server_negotiation.cc:299] Negotiation successful
0925 12:16:01.250590 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 12:16:01.252113 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:01.261895 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:01.253866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54985 (local address 127.0.0.1:38287)
0925 12:16:01.253903 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:01.253905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:01.253914 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:16:01.253986 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:01.253986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:01.253996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:01.254069 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:01.254775 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:01.255585 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:01.256731 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:01.257104 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:01.257168 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:01.258676 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:01.258677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:01.258678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:01.258685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:01.260397 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:01.260931 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:01.260932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:01.260933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:01.260992 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:01.261245 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:01.261245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:01.261246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:01.261370 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:01.261469 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:01.261632 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 12:16:01.261699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:16:01.278736 20705 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:01.279583 20705 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:01.284718 20705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aae4d6a6979490b9135879183916b6f: Generated new TSK 0
I20260925 12:16:01.284941 20705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:01.295135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20623
W20260925 12:16:01.299798 20608 connection.cc:535] client connection to 127.0.0.1:38287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38287): Connection reset by peer (error 104)
2026-09-25T12:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20722](info): set up 2 sockets
Sep 25 12:16:01 dist-test-slave-2hqq krb5kdc[20722](info): commencing operation
krb5kdc: starting...
W20260925 12:16:02.332624 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:02 dist-test-slave-2hqq krb5kdc[20722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:02.451759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:02.532955 20735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:02.533159 20735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:02.533182 20735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:02.534343 20735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:02.534385 20735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:02.534399 20735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:02.534449 20735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:02.534471 20735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:02.535959 20735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:02.536202 20735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:02.536427 20735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:02.541235 20743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:02.541275 20740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:02.541344 20741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:02.541440 20735 server_base.cc:1034] running on GCE node
I20260925 12:16:02.541762 20735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:02.542039 20735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:02.543190 20735 hybrid_clock.cc:648] HybridClock initialized: now 1790338562543168 us; error 36 us; skew 500 ppm
Sep 25 12:16:02 dist-test-slave-2hqq krb5kdc[20722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:02.545862 20735 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:02.546290 20735 webserver.cc:460] Webserver started at http://127.0.0.1:40983/ using document root <none> and password file <none>
I20260925 12:16:02.546523 20735 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:02.546587 20735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:02.546732 20735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:02.547814 20735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "10529598a1d84e6687058ac07138041f"
format_stamp: "Formatted at 2026-09-25 12:16:02 on dist-test-slave-2hqq"
server_key: "7de99cc28db958e6b93970c4fe0b4225"
server_key_iv: "59b28de2f3037965ff9956bd97064fe0"
server_key_version: "encryptionkey@0"
I20260925 12:16:02.548175 20735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "10529598a1d84e6687058ac07138041f"
format_stamp: "Formatted at 2026-09-25 12:16:02 on dist-test-slave-2hqq"
server_key: "7de99cc28db958e6b93970c4fe0b4225"
server_key_iv: "59b28de2f3037965ff9956bd97064fe0"
server_key_version: "encryptionkey@0"
I20260925 12:16:02.549563 20735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:16:02.550330 20750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:02.550580 20735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:16:02.550693 20735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "10529598a1d84e6687058ac07138041f"
format_stamp: "Formatted at 2026-09-25 12:16:02 on dist-test-slave-2hqq"
server_key: "7de99cc28db958e6b93970c4fe0b4225"
server_key_iv: "59b28de2f3037965ff9956bd97064fe0"
server_key_version: "encryptionkey@0"
I20260925 12:16:02.550803 20735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:02.564158 20735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:02.566179 20735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:02.566341 20735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:02.570253 20735 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39133
I20260925 12:16:02.570307 20802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39133 every 8 connection(s)
I20260925 12:16:02.570851 20735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:02.571700 20803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:02.574101 20803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f: Bootstrap starting.
I20260925 12:16:02.574685 20803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:02.574970 20803 log.cc:824] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:02.575654 20803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f: No bootstrap required, opened a new log
I20260925 12:16:02.576646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20735
I20260925 12:16:02.576766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:02.576972 392 external_mini_cluster.cc:1428] Setting key 57c3b6e8a79372cc93135aeed421680f
I20260925 12:16:02.577755 20803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260925 12:16:02.577937 20803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:02.577972 20803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10529598a1d84e6687058ac07138041f, State: Initialized, Role: FOLLOWER
I20260925 12:16:02.578099 20803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260925 12:16:02.578187 20803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:02.578222 20803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:02.578269 20803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:02.579047 20803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260925 12:16:02.579155 20803 leader_election.cc:302] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10529598a1d84e6687058ac07138041f; no voters:
I20260925 12:16:02.579303 20803 leader_election.cc:288] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:02.579351 20808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:02.579490 20808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [term 1 LEADER]: Becoming Leader. State: Replica: 10529598a1d84e6687058ac07138041f, State: Running, Role: LEADER
I20260925 12:16:02.579547 20803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:02.579614 20808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260925 12:16:02.580025 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10529598a1d84e6687058ac07138041f. Latest consensus state: current_term: 1 leader_uuid: "10529598a1d84e6687058ac07138041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } } }
I20260925 12:16:02.580252 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:02.580165 20810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10529598a1d84e6687058ac07138041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10529598a1d84e6687058ac07138041f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } } }
I20260925 12:16:02.580677 20810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:02.580696 20817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:02.581255 20817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:02.582991 20817 catalog_manager.cc:1347] Generated new cluster ID: 8b4b840e3ba9430bba0745161f208965
I20260925 12:16:02.583043 20817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:02.583494 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:02.577509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37055 (local address 127.0.0.1:39133)
0925 12:16:02.577728 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:02.577737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:02.577772 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:16:02.577886 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:02.577893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:02.577950 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:02.578093 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:02.578333 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:02.580096 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:02.581377 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:02.581948 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:02.582058 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:02.582774 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:02.582778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:02.582778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:02.583033 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:02.583075 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:02.583090 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:02.583170 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:02.583279 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 12:16:02.583305 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260925 12:16:02.584398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:02.589398 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:02.585898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40399 (local address 127.0.0.1:39133)
0925 12:16:02.585939 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:02.585941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:02.586024 (+ 83us) server_negotiation.cc:407] Connection header received
0925 12:16:02.586110 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:02.586110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:02.586120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:02.586146 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:02.586315 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:02.587030 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:02.588037 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:02.588344 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:02.588380 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:02.589146 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:02.589147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:02.589147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:02.589152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:02.589165 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:02.589168 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:02.589196 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:02.589310 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 12:16:02.589323 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:16:02.596843 20817 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:02.597358 20817 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:02.609647 20817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10529598a1d84e6687058ac07138041f: Generated new TSK 0
I20260925 12:16:02.609884 20817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:02.611950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20735
W20260925 12:16:02.616009 20720 connection.cc:535] client connection to 127.0.0.1:39133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39133): Connection reset by peer (error 104)
2026-09-25T12:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:02 dist-test-slave-2hqq krb5kdc[20834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:02 dist-test-slave-2hqq krb5kdc[20834](info): set up 2 sockets
Sep 25 12:16:02 dist-test-slave-2hqq krb5kdc[20834](info): commencing operation
krb5kdc: starting...
W20260925 12:16:03.648648 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:03 dist-test-slave-2hqq krb5kdc[20834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:03.769649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:03.851289 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:03.851516 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:03.851548 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:03.852763 20847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:03.852811 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:03.852825 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:03.852838 20847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:03.852895 20847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:03.854409 20847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:03.854686 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:03.854923 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:03.859891 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:03.859901 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:03.860107 20855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:03.860095 20847 server_base.cc:1034] running on GCE node
I20260925 12:16:03.860379 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:03.860679 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:03.861845 20847 hybrid_clock.cc:648] HybridClock initialized: now 1790338563861819 us; error 30 us; skew 500 ppm
Sep 25 12:16:03 dist-test-slave-2hqq krb5kdc[20834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:03.864737 20847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:03.865222 20847 webserver.cc:460] Webserver started at http://127.0.0.1:36419/ using document root <none> and password file <none>
I20260925 12:16:03.865450 20847 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:03.865494 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:03.865598 20847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:03.866739 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7"
format_stamp: "Formatted at 2026-09-25 12:16:03 on dist-test-slave-2hqq"
server_key: "89473dd5fe9d0bf2f4113054db10fed9"
server_key_iv: "96113a7a661bd500918e92f3fb6bdfda"
server_key_version: "encryptionkey@0"
I20260925 12:16:03.867074 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7"
format_stamp: "Formatted at 2026-09-25 12:16:03 on dist-test-slave-2hqq"
server_key: "89473dd5fe9d0bf2f4113054db10fed9"
server_key_iv: "96113a7a661bd500918e92f3fb6bdfda"
server_key_version: "encryptionkey@0"
I20260925 12:16:03.868819 20847 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260925 12:16:03.869905 20862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:03.870118 20847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:03.870229 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7"
format_stamp: "Formatted at 2026-09-25 12:16:03 on dist-test-slave-2hqq"
server_key: "89473dd5fe9d0bf2f4113054db10fed9"
server_key_iv: "96113a7a661bd500918e92f3fb6bdfda"
server_key_version: "encryptionkey@0"
I20260925 12:16:03.870324 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:03.876107 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:03.878655 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:03.878778 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:03.883023 20847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37869
I20260925 12:16:03.883059 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37869 every 8 connection(s)
I20260925 12:16:03.883410 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:03.883992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20847
I20260925 12:16:03.884171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:03.884414 392 external_mini_cluster.cc:1428] Setting key a36d17ffd4b721d8de3b1a7ef13ad4f3
I20260925 12:16:03.884687 20915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:03.886979 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:03.884894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45935 (local address 127.0.0.1:37869)
0925 12:16:03.885060 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:03.885071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:03.885108 (+ 37us) server_negotiation.cc:407] Connection header received
0925 12:16:03.885577 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:03.885584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:03.885617 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:03.885773 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:03.886522 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:03.886526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:03.886526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:03.886530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:03.886579 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:03.886589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:03.886665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:03.886736 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:16:03.886779 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260925 12:16:03.887952 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7: Bootstrap starting.
I20260925 12:16:03.888000 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:03.888690 20915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:03.889022 20915 log.cc:824] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:03.889864 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7: No bootstrap required, opened a new log
I20260925 12:16:03.890769 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:03.889673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44643 (local address 127.0.0.1:37869)
0925 12:16:03.889709 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:03.889711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:03.889901 (+ 190us) server_negotiation.cc:407] Connection header received
0925 12:16:03.889956 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:03.889957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:03.889962 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:03.890029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:03.890552 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:03.890553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:03.890554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:03.890557 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:03.890568 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:03.890571 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:03.890615 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:03.890669 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 12:16:03.890679 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:16:03.892133 20915 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260925 12:16:03.892268 20915 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:03.892290 20915 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f63bfdc56a4b59b0284ecd4f5135e7, State: Initialized, Role: FOLLOWER
I20260925 12:16:03.892402 20915 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260925 12:16:03.892527 20915 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:03.892560 20915 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:03.892599 20915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:03.893087 20915 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260925 12:16:03.893175 20915 leader_election.cc:302] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1f63bfdc56a4b59b0284ecd4f5135e7; no voters:
I20260925 12:16:03.893311 20915 leader_election.cc:288] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:03.893380 20925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:03.893548 20915 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:03.893559 20925 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [term 1 LEADER]: Becoming Leader. State: Replica: f1f63bfdc56a4b59b0284ecd4f5135e7, State: Running, Role: LEADER
I20260925 12:16:03.893671 20925 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260925 12:16:03.894064 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1f63bfdc56a4b59b0284ecd4f5135e7. Latest consensus state: current_term: 1 leader_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } } }
I20260925 12:16:03.894186 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:03.894410 20934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:03.894057 20926 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f63bfdc56a4b59b0284ecd4f5135e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } } }
I20260925 12:16:03.894732 20926 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:03.894992 20934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:03.896363 20934 catalog_manager.cc:1347] Generated new cluster ID: af3422c7aab34884bed4a4b5b39c1011
I20260925 12:16:03.896413 20934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:03.903409 20934 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:03.903568 20934 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:03.908354 20934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1f63bfdc56a4b59b0284ecd4f5135e7: Generated new TSK 0
I20260925 12:16:03.908561 20934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:03.912077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20847
2026-09-25T12:16:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:03 dist-test-slave-2hqq krb5kdc[20946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:03 dist-test-slave-2hqq krb5kdc[20946](info): set up 2 sockets
Sep 25 12:16:03 dist-test-slave-2hqq krb5kdc[20946](info): commencing operation
krb5kdc: starting...
W20260925 12:16:04.948719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:05.068778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:05.150877 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:05.151168 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:05.151206 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:05.152433 20959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:05.152479 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:05.152523 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:05.152572 20959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:05.152601 20959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:05.154155 20959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:05.154410 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:05.154659 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:05.159754 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:05.159945 20959 server_base.cc:1034] running on GCE node
W20260925 12:16:05.159686 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:05.159683 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:05.160351 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:05.160722 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:05.161895 20959 hybrid_clock.cc:648] HybridClock initialized: now 1790338565161887 us; error 39 us; skew 500 ppm
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:05.164584 20959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:05.165076 20959 webserver.cc:460] Webserver started at http://127.0.0.1:39093/ using document root <none> and password file <none>
I20260925 12:16:05.165334 20959 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:05.165398 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:05.165540 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:05.166604 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "4244da4b8a7d48ee888b318b76b8ec2e"
format_stamp: "Formatted at 2026-09-25 12:16:05 on dist-test-slave-2hqq"
server_key: "e10403c5bf259148becc53796b29f1b7"
server_key_iv: "08cd3a70c4f602d6f856fac9a8d79fce"
server_key_version: "encryptionkey@0"
I20260925 12:16:05.166956 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "4244da4b8a7d48ee888b318b76b8ec2e"
format_stamp: "Formatted at 2026-09-25 12:16:05 on dist-test-slave-2hqq"
server_key: "e10403c5bf259148becc53796b29f1b7"
server_key_iv: "08cd3a70c4f602d6f856fac9a8d79fce"
server_key_version: "encryptionkey@0"
I20260925 12:16:05.168272 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:05.169094 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:05.169274 20959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:05.169370 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "4244da4b8a7d48ee888b318b76b8ec2e"
format_stamp: "Formatted at 2026-09-25 12:16:05 on dist-test-slave-2hqq"
server_key: "e10403c5bf259148becc53796b29f1b7"
server_key_iv: "08cd3a70c4f602d6f856fac9a8d79fce"
server_key_version: "encryptionkey@0"
I20260925 12:16:05.169452 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:05.202333 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:05.204735 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:05.204902 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:05.209103 20959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37027
I20260925 12:16:05.209144 21026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37027 every 8 connection(s)
I20260925 12:16:05.209540 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:05.210454 21027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:05.212853 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e: Bootstrap starting.
I20260925 12:16:05.213457 21027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:05.213752 21027 log.cc:824] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:05.214051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 20959
I20260925 12:16:05.214177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:05.214371 392 external_mini_cluster.cc:1428] Setting key cb2e29ef950fbb6294e679534103db9d
I20260925 12:16:05.214833 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e: No bootstrap required, opened a new log
I20260925 12:16:05.217377 21027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260925 12:16:05.217528 21027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:05.217552 21027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4244da4b8a7d48ee888b318b76b8ec2e, State: Initialized, Role: FOLLOWER
I20260925 12:16:05.217700 21027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260925 12:16:05.217764 21027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:05.217804 21027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:05.217855 21027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:05.218454 21027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260925 12:16:05.218564 21027 leader_election.cc:302] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4244da4b8a7d48ee888b318b76b8ec2e; no voters:
I20260925 12:16:05.218784 21027 leader_election.cc:288] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:05.218829 21032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:05.218999 21032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [term 1 LEADER]: Becoming Leader. State: Replica: 4244da4b8a7d48ee888b318b76b8ec2e, State: Running, Role: LEADER
I20260925 12:16:05.219055 21027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:05.219169 21032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } }
I20260925 12:16:05.219584 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4244da4b8a7d48ee888b318b76b8ec2e. Latest consensus state: current_term: 1 leader_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } } }
I20260925 12:16:05.219595 21034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4244da4b8a7d48ee888b318b76b8ec2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37027 } } }
I20260925 12:16:05.219739 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:05.219748 21034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:05.219971 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:05.220659 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:05.222448 21041 catalog_manager.cc:1347] Generated new cluster ID: 907b8c01371f40fcb75bb7923daba4b2
I20260925 12:16:05.222520 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[20946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:16:05.227505 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:05.214911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55767 (local address 127.0.0.1:37027)
0925 12:16:05.215096 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:05.215104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:05.215133 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:16:05.216185 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:05.216192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:05.216260 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:05.216392 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:05.217311 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:05.218683 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:05.219858 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:05.220601 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:05.220715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:05.223663 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:05.223667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:05.223668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:05.223681 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:05.225773 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:05.226190 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:05.226193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:05.226193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:05.226314 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:05.226512 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:05.226512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:05.226513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:05.226853 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:05.227020 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:05.227106 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 12:16:05.227262 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 12:16:05.228672 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:05.235267 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:05.236078 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:05.238147 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:05.230172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34955 (local address 127.0.0.1:37027)
0925 12:16:05.230215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:05.230217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:05.230315 (+ 98us) server_negotiation.cc:407] Connection header received
0925 12:16:05.230360 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:05.230361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:05.230374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:05.230444 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:05.231050 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:05.231851 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:05.232976 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:05.233320 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:05.233361 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:05.235333 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:05.235334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:05.235334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:05.235341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:05.237008 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:05.237399 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:05.237401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:05.237401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:05.237458 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:05.237648 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:05.237648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:05.237649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:05.237762 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:05.237846 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:05.237974 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 12:16:05.238059 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":22}
I20260925 12:16:05.239354 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4244da4b8a7d48ee888b318b76b8ec2e: Generated new TSK 0
I20260925 12:16:05.239528 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:05.241899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 20959
2026-09-25T12:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[21058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[21058](info): set up 2 sockets
Sep 25 12:16:05 dist-test-slave-2hqq krb5kdc[21058](info): commencing operation
krb5kdc: starting...
W20260925 12:16:06.284624 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:06 dist-test-slave-2hqq krb5kdc[21058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:06.407787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:06.494478 21071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:06.494752 21071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:06.494787 21071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:06.495949 21071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:06.496024 21071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:06.496047 21071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:06.496068 21071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:06.496119 21071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:06.497665 21071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:06.497942 21071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:06.498176 21071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:06.503214 21077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:06.503227 21076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:06.503463 21071 server_base.cc:1034] running on GCE node
W20260925 12:16:06.503542 21079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:06.503759 21071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:06.504067 21071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:06.505276 21071 hybrid_clock.cc:648] HybridClock initialized: now 1790338566505266 us; error 33 us; skew 500 ppm
Sep 25 12:16:06 dist-test-slave-2hqq krb5kdc[21058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:06.513808 21071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:06.514429 21071 webserver.cc:460] Webserver started at http://127.0.0.1:42247/ using document root <none> and password file <none>
I20260925 12:16:06.514688 21071 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:06.514758 21071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:06.514899 21071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:06.516029 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec"
format_stamp: "Formatted at 2026-09-25 12:16:06 on dist-test-slave-2hqq"
server_key: "f664e9f5f197bbbfcef89238706b0a3a"
server_key_iv: "b2aa9f0461ad6ce41cfa9ae8c9522149"
server_key_version: "encryptionkey@0"
I20260925 12:16:06.516405 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec"
format_stamp: "Formatted at 2026-09-25 12:16:06 on dist-test-slave-2hqq"
server_key: "f664e9f5f197bbbfcef89238706b0a3a"
server_key_iv: "b2aa9f0461ad6ce41cfa9ae8c9522149"
server_key_version: "encryptionkey@0"
I20260925 12:16:06.517823 21071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:16:06.518615 21086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:06.518787 21071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:06.518891 21071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec"
format_stamp: "Formatted at 2026-09-25 12:16:06 on dist-test-slave-2hqq"
server_key: "f664e9f5f197bbbfcef89238706b0a3a"
server_key_iv: "b2aa9f0461ad6ce41cfa9ae8c9522149"
server_key_version: "encryptionkey@0"
I20260925 12:16:06.518963 21071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:06.527314 21071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:06.529470 21071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:06.529636 21071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:06.533686 21071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45377
I20260925 12:16:06.533730 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45377 every 8 connection(s)
I20260925 12:16:06.534107 21071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:06.535112 21139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:06.537475 21139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec: Bootstrap starting.
I20260925 12:16:06.538112 21139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:06.538388 21139 log.cc:824] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:06.539130 21139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec: No bootstrap required, opened a new log
I20260925 12:16:06.540839 21139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 12:16:06.540978 21139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:06.541000 21139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd66521b40945b8a4d5ad0bcb9f44ec, State: Initialized, Role: FOLLOWER
I20260925 12:16:06.541157 21139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 12:16:06.541217 21139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:06.541237 21139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:06.541307 21139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:06.541847 21139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 12:16:06.541956 21139 leader_election.cc:302] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fd66521b40945b8a4d5ad0bcb9f44ec; no voters:
I20260925 12:16:06.542128 21139 leader_election.cc:288] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:06.542158 21142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:06.542404 21142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [term 1 LEADER]: Becoming Leader. State: Replica: 1fd66521b40945b8a4d5ad0bcb9f44ec, State: Running, Role: LEADER
I20260925 12:16:06.542402 21139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:06.542558 21142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } }
I20260925 12:16:06.542935 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fd66521b40945b8a4d5ad0bcb9f44ec. Latest consensus state: current_term: 1 leader_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260925 12:16:06.542979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21071
I20260925 12:16:06.543023 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:06.543119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:06.543391 392 external_mini_cluster.cc:1428] Setting key dc4ec3dfdbbd9195e4d2b8125a412010
I20260925 12:16:06.543267 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd66521b40945b8a4d5ad0bcb9f44ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45377 } } }
I20260925 12:16:06.543536 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:06.543463 21146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:06.544045 21146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:06.546890 21146 catalog_manager.cc:1347] Generated new cluster ID: 91ca075e1c364519930293d56cfb8e0c
I20260925 12:16:06.546948 21146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:06.549857 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:06.544052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43681 (local address 127.0.0.1:45377)
0925 12:16:06.544380 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:06.544390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:06.544693 (+ 303us) server_negotiation.cc:407] Connection header received
0925 12:16:06.544827 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:06.544833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:06.544878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:06.545022 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:06.545254 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:06.546732 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:06.547656 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:06.548243 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:06.548363 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:06.549120 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:06.549124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:06.549124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:06.549310 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:06.549359 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:06.549371 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:06.549448 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:06.549524 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 12:16:06.549688 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":68,"threads_started":1}
I20260925 12:16:06.550912 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:06.560586 21146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:06.561286 21146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:06.562634 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:06.559300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52641 (local address 127.0.0.1:45377)
0925 12:16:06.559370 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:06.559372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:06.559484 (+ 112us) server_negotiation.cc:407] Connection header received
0925 12:16:06.559554 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:06.559555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:06.559567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:06.559636 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:06.559779 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:06.560357 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:06.561235 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:06.561521 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:06.561599 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:06.562303 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:06.562305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:06.562305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:06.562311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:06.562350 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:06.562356 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:06.562415 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:06.562484 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 12:16:06.562529 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:16:06.566481 21146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fd66521b40945b8a4d5ad0bcb9f44ec: Generated new TSK 0
I20260925 12:16:06.566685 21146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:06.570266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21071
2026-09-25T12:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:06 dist-test-slave-2hqq krb5kdc[21170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:06 dist-test-slave-2hqq krb5kdc[21170](info): set up 2 sockets
Sep 25 12:16:06 dist-test-slave-2hqq krb5kdc[21170](info): commencing operation
krb5kdc: starting...
W20260925 12:16:07.608753 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:07 dist-test-slave-2hqq krb5kdc[21170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:07.727986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:07.809932 21183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:07.810185 21183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:07.810222 21183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:07.811401 21183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:07.811473 21183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:07.811496 21183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:07.811515 21183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:07.811547 21183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:07.813057 21183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:07.813336 21183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:07.813568 21183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:07.818454 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:07.818485 21188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:07.818679 21183 server_base.cc:1034] running on GCE node
W20260925 12:16:07.818513 21191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:07.819097 21183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:07.819356 21183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:07.820547 21183 hybrid_clock.cc:648] HybridClock initialized: now 1790338567820524 us; error 39 us; skew 500 ppm
Sep 25 12:16:07 dist-test-slave-2hqq krb5kdc[21170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:07.823088 21183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:07.823567 21183 webserver.cc:460] Webserver started at http://127.0.0.1:41107/ using document root <none> and password file <none>
I20260925 12:16:07.823814 21183 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:07.823885 21183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:07.824029 21183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:07.825157 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "df02a296b92b4020b3576f4eb76b96e3"
format_stamp: "Formatted at 2026-09-25 12:16:07 on dist-test-slave-2hqq"
server_key: "0d5e23287655d4d07e67f3b8787cc612"
server_key_iv: "c5fb8eaaa8bf3b2a281d998ef77f572c"
server_key_version: "encryptionkey@0"
I20260925 12:16:07.825523 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "df02a296b92b4020b3576f4eb76b96e3"
format_stamp: "Formatted at 2026-09-25 12:16:07 on dist-test-slave-2hqq"
server_key: "0d5e23287655d4d07e67f3b8787cc612"
server_key_iv: "c5fb8eaaa8bf3b2a281d998ef77f572c"
server_key_version: "encryptionkey@0"
I20260925 12:16:07.826859 21183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:16:07.827616 21198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:07.827780 21183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:16:07.827870 21183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "df02a296b92b4020b3576f4eb76b96e3"
format_stamp: "Formatted at 2026-09-25 12:16:07 on dist-test-slave-2hqq"
server_key: "0d5e23287655d4d07e67f3b8787cc612"
server_key_iv: "c5fb8eaaa8bf3b2a281d998ef77f572c"
server_key_version: "encryptionkey@0"
I20260925 12:16:07.827944 21183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:07.831362 21183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:07.833405 21183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:07.833580 21183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:07.837618 21183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37409
I20260925 12:16:07.837654 21250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37409 every 8 connection(s)
I20260925 12:16:07.838032 21183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:07.839002 21251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:07.841500 21251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3: Bootstrap starting.
I20260925 12:16:07.842072 21251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:07.842352 21251 log.cc:824] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:07.842903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21183
I20260925 12:16:07.843024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:07.843169 21251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3: No bootstrap required, opened a new log
I20260925 12:16:07.843230 392 external_mini_cluster.cc:1428] Setting key 277409025c7ffefa544dd9925256ec38
I20260925 12:16:07.845665 21251 raft_consensus.cc:348] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260925 12:16:07.845870 21251 raft_consensus.cc:374] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:07.845919 21251 raft_consensus.cc:729] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df02a296b92b4020b3576f4eb76b96e3, State: Initialized, Role: FOLLOWER
I20260925 12:16:07.846060 21251 consensus_queue.cc:260] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260925 12:16:07.846153 21251 raft_consensus.cc:388] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:07.846182 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:07.843768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43305 (local address 127.0.0.1:37409)
0925 12:16:07.844013 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:07.844022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:07.844048 (+ 26us) server_negotiation.cc:407] Connection header received
0925 12:16:07.844595 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:07.844603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:07.844631 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:07.844756 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:07.845474 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:07.845477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:07.845478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:07.845482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:07.845520 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:07.845530 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:07.845588 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:07.845678 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 12:16:07.845733 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260925 12:16:07.846196 21251 raft_consensus.cc:482] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:07.846273 21251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:07.846958 21251 raft_consensus.cc:504] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260925 12:16:07.847082 21251 leader_election.cc:302] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df02a296b92b4020b3576f4eb76b96e3; no voters:
I20260925 12:16:07.847151 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:07.847262 21251 leader_election.cc:288] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:07.847285 21256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:07.847390 21256 raft_consensus.cc:686] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [term 1 LEADER]: Becoming Leader. State: Replica: df02a296b92b4020b3576f4eb76b96e3, State: Running, Role: LEADER
I20260925 12:16:07.847520 21251 sys_catalog.cc:564] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:07.847539 21256 consensus_queue.cc:237] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } }
I20260925 12:16:07.847990 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df02a296b92b4020b3576f4eb76b96e3. Latest consensus state: current_term: 1 leader_uuid: "df02a296b92b4020b3576f4eb76b96e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } } }
I20260925 12:16:07.848256 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:07.848166 21257 sys_catalog.cc:455] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df02a296b92b4020b3576f4eb76b96e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02a296b92b4020b3576f4eb76b96e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37409 } } }
I20260925 12:16:07.848713 21257 sys_catalog.cc:458] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:07.848873 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:07.849372 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:07.850864 21268 catalog_manager.cc:1347] Generated new cluster ID: 7a1edcdc897541a894e5f525c60490ab
I20260925 12:16:07.850919 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:07.853633 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:07.850503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53111 (local address 127.0.0.1:37409)
0925 12:16:07.850543 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:07.850545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:07.852686 (+ 2141us) server_negotiation.cc:407] Connection header received
0925 12:16:07.852753 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:07.852754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:07.852759 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:07.852822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:07.853431 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:07.853432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:07.853432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:07.853436 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:07.853451 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:07.853455 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:07.853505 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:07.853561 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 12:16:07.853573 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:16:07.883255 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:07.883471 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:07.895502 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df02a296b92b4020b3576f4eb76b96e3: Generated new TSK 0
I20260925 12:16:07.895751 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:07.901559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21183
2026-09-25T12:16:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:07 dist-test-slave-2hqq krb5kdc[21282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:07 dist-test-slave-2hqq krb5kdc[21282](info): set up 2 sockets
Sep 25 12:16:07 dist-test-slave-2hqq krb5kdc[21282](info): commencing operation
krb5kdc: starting...
W20260925 12:16:08.944746 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:09.068272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:09.151197 21295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:09.151479 21295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:09.151520 21295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:09.152782 21295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:09.152845 21295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:09.152877 21295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:09.152909 21295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:09.152935 21295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:09.154480 21295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:09.154754 21295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:09.154980 21295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:09.159781 21303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:09.159781 21300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:09.159776 21301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:09.159968 21295 server_base.cc:1034] running on GCE node
I20260925 12:16:09.160296 21295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:09.160610 21295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:09.161758 21295 hybrid_clock.cc:648] HybridClock initialized: now 1790338569161747 us; error 27 us; skew 500 ppm
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:09.164260 21295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:09.164759 21295 webserver.cc:460] Webserver started at http://127.0.0.1:42429/ using document root <none> and password file <none>
I20260925 12:16:09.164999 21295 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:09.165063 21295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:09.165201 21295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:09.166323 21295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "4cee7eeedea4480a81c1c16e7b752731"
format_stamp: "Formatted at 2026-09-25 12:16:09 on dist-test-slave-2hqq"
server_key: "61dd19573ccbe0b7b27c5053ab58d490"
server_key_iv: "9135b55bda9d37ea5eb1757c39847726"
server_key_version: "encryptionkey@0"
I20260925 12:16:09.166716 21295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cee7eeedea4480a81c1c16e7b752731"
format_stamp: "Formatted at 2026-09-25 12:16:09 on dist-test-slave-2hqq"
server_key: "61dd19573ccbe0b7b27c5053ab58d490"
server_key_iv: "9135b55bda9d37ea5eb1757c39847726"
server_key_version: "encryptionkey@0"
I20260925 12:16:09.168129 21295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:09.168974 21310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:09.169165 21295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:09.169265 21295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "4cee7eeedea4480a81c1c16e7b752731"
format_stamp: "Formatted at 2026-09-25 12:16:09 on dist-test-slave-2hqq"
server_key: "61dd19573ccbe0b7b27c5053ab58d490"
server_key_iv: "9135b55bda9d37ea5eb1757c39847726"
server_key_version: "encryptionkey@0"
I20260925 12:16:09.169345 21295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:09.197825 21295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:09.199847 21295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:09.200013 21295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:09.204181 21295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41927
I20260925 12:16:09.204221 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41927 every 8 connection(s)
I20260925 12:16:09.204643 21295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:09.205566 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:09.208000 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731: Bootstrap starting.
I20260925 12:16:09.208702 21363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:09.208989 21363 log.cc:824] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:09.209743 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731: No bootstrap required, opened a new log
I20260925 12:16:09.211308 21363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } }
I20260925 12:16:09.211448 21363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:09.211470 21363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cee7eeedea4480a81c1c16e7b752731, State: Initialized, Role: FOLLOWER
I20260925 12:16:09.211665 21363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } }
I20260925 12:16:09.211746 21363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:09.211794 21363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:09.211848 21363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:09.212396 21363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } }
I20260925 12:16:09.212534 21363 leader_election.cc:302] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cee7eeedea4480a81c1c16e7b752731; no voters:
I20260925 12:16:09.212714 21363 leader_election.cc:288] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:09.212749 21366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:09.212927 21366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [term 1 LEADER]: Becoming Leader. State: Replica: 4cee7eeedea4480a81c1c16e7b752731, State: Running, Role: LEADER
I20260925 12:16:09.212972 21363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:09.213013 21366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } }
I20260925 12:16:09.213418 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cee7eeedea4480a81c1c16e7b752731. Latest consensus state: current_term: 1 leader_uuid: "4cee7eeedea4480a81c1c16e7b752731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } } }
I20260925 12:16:09.213532 21367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cee7eeedea4480a81c1c16e7b752731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cee7eeedea4480a81c1c16e7b752731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41927 } } }
I20260925 12:16:09.213555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21295
I20260925 12:16:09.213560 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:09.213678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:09.213688 21367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:09.213873 21375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:09.213888 392 external_mini_cluster.cc:1428] Setting key 4bf7337d16e1ca9d98567a798172feba
I20260925 12:16:09.214403 21375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:09.216164 21375 catalog_manager.cc:1347] Generated new cluster ID: 021acaa7887e408f984c248815166df5
I20260925 12:16:09.216216 21375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:16:09.223893 21375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:09.224390 21375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:09.225957 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:09.214542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:50737 (local address 127.0.0.1:41927)
0925 12:16:09.214787 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:09.214798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:09.214899 (+ 101us) server_negotiation.cc:407] Connection header received
0925 12:16:09.215074 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:09.215082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:09.215152 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:09.215308 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:09.216136 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:09.217563 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:09.218695 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:09.219306 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:09.219396 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:09.222229 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:09.222233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:09.222234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:09.222245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:09.224249 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:09.224743 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:09.224746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:09.224747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:09.224856 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:09.225084 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:09.225085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:09.225086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:09.225376 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:09.225532 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:09.225640 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:16:09.225746 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260925 12:16:09.226912 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:09.228998 21375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cee7eeedea4480a81c1c16e7b752731: Generated new TSK 0
I20260925 12:16:09.229174 21375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:09.235049 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:09.228452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:41283 (local address 127.0.0.1:41927)
0925 12:16:09.228530 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:09.228532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:09.228590 (+ 58us) server_negotiation.cc:407] Connection header received
0925 12:16:09.228653 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:09.228655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:09.228664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:09.228722 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:09.229385 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:09.229892 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:09.230933 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:09.231133 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:09.231180 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:09.232561 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:09.232562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:09.232562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:09.232569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:09.233937 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:09.234338 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:09.234339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:09.234339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:09.234386 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:09.234598 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:09.234598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:09.234598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:09.234715 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:09.234808 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:09.234948 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 12:16:09.234993 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 12:16:09.237224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21295
W20260925 12:16:09.241873 21280 connection.cc:535] client connection to 127.0.0.1:41927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41927): Connection reset by peer (error 104)
2026-09-25T12:16:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21394](info): set up 2 sockets
Sep 25 12:16:09 dist-test-slave-2hqq krb5kdc[21394](info): commencing operation
krb5kdc: starting...
W20260925 12:16:10.292826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:10 dist-test-slave-2hqq krb5kdc[21394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:10.412091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:10.493232 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:10.493512 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:10.493551 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:10.494736 21407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:10.494812 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:10.494838 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:10.494859 21407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:10.494885 21407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:10.496403 21407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:10.496706 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:10.496940 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:10.501857 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:10.501930 21407 server_base.cc:1034] running on GCE node
W20260925 12:16:10.501858 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:10.501852 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:10.502295 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:10.502605 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:10.503767 21407 hybrid_clock.cc:648] HybridClock initialized: now 1790338570503740 us; error 42 us; skew 500 ppm
Sep 25 12:16:10 dist-test-slave-2hqq krb5kdc[21394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:10.506374 21407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:10.506868 21407 webserver.cc:460] Webserver started at http://127.0.0.1:34641/ using document root <none> and password file <none>
I20260925 12:16:10.507102 21407 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:10.507166 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:10.507319 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:10.508426 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "3549704edf8c494c9a753b8893962c72"
format_stamp: "Formatted at 2026-09-25 12:16:10 on dist-test-slave-2hqq"
server_key: "3fd58212d2bf85a38afa28343ff18753"
server_key_iv: "b161e5779af3ce1de482fbc5f6885033"
server_key_version: "encryptionkey@0"
I20260925 12:16:10.508849 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "3549704edf8c494c9a753b8893962c72"
format_stamp: "Formatted at 2026-09-25 12:16:10 on dist-test-slave-2hqq"
server_key: "3fd58212d2bf85a38afa28343ff18753"
server_key_iv: "b161e5779af3ce1de482fbc5f6885033"
server_key_version: "encryptionkey@0"
I20260925 12:16:10.510165 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:10.510840 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:10.511008 21407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:16:10.511101 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "3549704edf8c494c9a753b8893962c72"
format_stamp: "Formatted at 2026-09-25 12:16:10 on dist-test-slave-2hqq"
server_key: "3fd58212d2bf85a38afa28343ff18753"
server_key_iv: "b161e5779af3ce1de482fbc5f6885033"
server_key_version: "encryptionkey@0"
I20260925 12:16:10.511183 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:10.522181 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:10.524135 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:10.524305 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:10.528360 21407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37067
I20260925 12:16:10.528406 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37067 every 8 connection(s)
I20260925 12:16:10.528861 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:10.529836 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:10.532253 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: Bootstrap starting.
I20260925 12:16:10.532863 21475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:10.533143 21475 log.cc:824] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:10.533828 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: No bootstrap required, opened a new log
I20260925 12:16:10.535625 21475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260925 12:16:10.535779 21475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:10.535804 21475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3549704edf8c494c9a753b8893962c72, State: Initialized, Role: FOLLOWER
I20260925 12:16:10.535980 21475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260925 12:16:10.536044 21475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:10.536084 21475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:10.536136 21475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:10.536707 21475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260925 12:16:10.536794 21475 leader_election.cc:302] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3549704edf8c494c9a753b8893962c72; no voters:
I20260925 12:16:10.536954 21475 leader_election.cc:288] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:10.536984 21478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:10.537067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21407
I20260925 12:16:10.537108 21478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [term 1 LEADER]: Becoming Leader. State: Replica: 3549704edf8c494c9a753b8893962c72, State: Running, Role: LEADER
I20260925 12:16:10.537210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:10.537186 21478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260925 12:16:10.537237 21475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:10.537423 392 external_mini_cluster.cc:1428] Setting key 15ffa838f895af89a0d0021e15dbad79
I20260925 12:16:10.537763 21480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3549704edf8c494c9a753b8893962c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260925 12:16:10.537860 21480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:10.538025 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3549704edf8c494c9a753b8893962c72. Latest consensus state: current_term: 1 leader_uuid: "3549704edf8c494c9a753b8893962c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3549704edf8c494c9a753b8893962c72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260925 12:16:10.538079 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72 [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:10.539110 21496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:10.539168 21496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:10.539237 21488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:10.539378 21488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:10.540938 21488 catalog_manager.cc:1347] Generated new cluster ID: 6e9caecbca61471bbc68ef660d6a1fc7
I20260925 12:16:10.540997 21488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:10.543684 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:10.538194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:56769 (local address 127.0.0.1:37067)
0925 12:16:10.538396 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:10.538406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:10.538446 (+ 40us) server_negotiation.cc:407] Connection header received
0925 12:16:10.538600 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:10.538607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:10.538751 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:10.538927 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:10.539160 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:10.540375 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:10.541518 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:10.542107 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:10.542216 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:10.542983 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:10.542986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:10.542987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:10.543197 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:10.543233 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:10.543246 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:10.543324 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:10.543435 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 12:16:10.543461 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":122,"threads_started":1}
I20260925 12:16:10.544682 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:10.547189 21488 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:10.547966 21488 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:10.549862 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:10.546123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:35889 (local address 127.0.0.1:37067)
0925 12:16:10.546155 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:10.546157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:10.546401 (+ 244us) server_negotiation.cc:407] Connection header received
0925 12:16:10.546483 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:10.546483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:10.546504 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:10.546531 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:10.546750 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:10.547405 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:10.548522 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:10.548827 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:10.548886 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:10.549600 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:10.549601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:10.549602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:10.549606 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:10.549621 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:10.549626 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:10.549684 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:10.549782 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 12:16:10.549793 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 12:16:10.552416 21488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3549704edf8c494c9a753b8893962c72: Generated new TSK 0
I20260925 12:16:10.552624 21488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:10.557551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21407
W20260925 12:16:10.561646 21392 connection.cc:535] client connection to 127.0.0.1:37067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37067): Connection reset by peer (error 104)
2026-09-25T12:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:10 dist-test-slave-2hqq krb5kdc[21506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:10 dist-test-slave-2hqq krb5kdc[21506](info): set up 2 sockets
Sep 25 12:16:10 dist-test-slave-2hqq krb5kdc[21506](info): commencing operation
krb5kdc: starting...
W20260925 12:16:11.600663 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:11 dist-test-slave-2hqq krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:11.719666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:11.801792 21519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:11.802047 21519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:11.802088 21519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:11.803228 21519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:11.803303 21519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:11.803328 21519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:11.803354 21519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:11.803382 21519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:11.804889 21519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:11.805167 21519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:11.805393 21519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:16:11.810318 21519 server_base.cc:1034] running on GCE node
W20260925 12:16:11.810292 21525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:11.810292 21527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:11.810284 21524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:11.810770 21519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:11.811014 21519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:11.812155 21519 hybrid_clock.cc:648] HybridClock initialized: now 1790338571812141 us; error 29 us; skew 500 ppm
Sep 25 12:16:11 dist-test-slave-2hqq krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:11.820106 21519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:11.820653 21519 webserver.cc:460] Webserver started at http://127.0.0.1:39621/ using document root <none> and password file <none>
I20260925 12:16:11.820863 21519 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:11.820905 21519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:11.821008 21519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:11.822117 21519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "045264dd49b4442daf203c1749fb50fe"
format_stamp: "Formatted at 2026-09-25 12:16:11 on dist-test-slave-2hqq"
server_key: "8d3f4bddc21c4fae8806a189fc1d35a3"
server_key_iv: "98fdcad4439c5cdd5d8c9b4e2afb514a"
server_key_version: "encryptionkey@0"
I20260925 12:16:11.822417 21519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "045264dd49b4442daf203c1749fb50fe"
format_stamp: "Formatted at 2026-09-25 12:16:11 on dist-test-slave-2hqq"
server_key: "8d3f4bddc21c4fae8806a189fc1d35a3"
server_key_iv: "98fdcad4439c5cdd5d8c9b4e2afb514a"
server_key_version: "encryptionkey@0"
I20260925 12:16:11.823665 21519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:16:11.824348 21534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:11.824580 21519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 12:16:11.824673 21519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "045264dd49b4442daf203c1749fb50fe"
format_stamp: "Formatted at 2026-09-25 12:16:11 on dist-test-slave-2hqq"
server_key: "8d3f4bddc21c4fae8806a189fc1d35a3"
server_key_iv: "98fdcad4439c5cdd5d8c9b4e2afb514a"
server_key_version: "encryptionkey@0"
I20260925 12:16:11.824745 21519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:11.828014 21519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:11.830194 21519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:11.830345 21519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:11.834375 21519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40641
I20260925 12:16:11.834429 21586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40641 every 8 connection(s)
I20260925 12:16:11.834764 21519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:11.835714 21587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:11.837972 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe: Bootstrap starting.
I20260925 12:16:11.838564 21587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:11.838845 21587 log.cc:824] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:11.839524 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe: No bootstrap required, opened a new log
I20260925 12:16:11.841150 21587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260925 12:16:11.841290 21587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:11.841312 21587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045264dd49b4442daf203c1749fb50fe, State: Initialized, Role: FOLLOWER
I20260925 12:16:11.841502 21587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260925 12:16:11.841585 21587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:11.841638 21587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:11.841691 21587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:11.842206 21587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260925 12:16:11.842314 21587 leader_election.cc:302] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045264dd49b4442daf203c1749fb50fe; no voters:
I20260925 12:16:11.842496 21587 leader_election.cc:288] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:11.842542 21590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:11.842718 21590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [term 1 LEADER]: Becoming Leader. State: Replica: 045264dd49b4442daf203c1749fb50fe, State: Running, Role: LEADER
I20260925 12:16:11.842773 21587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:11.842844 21590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260925 12:16:11.843189 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045264dd49b4442daf203c1749fb50fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260925 12:16:11.843204 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045264dd49b4442daf203c1749fb50fe. Latest consensus state: current_term: 1 leader_uuid: "045264dd49b4442daf203c1749fb50fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045264dd49b4442daf203c1749fb50fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260925 12:16:11.843334 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:11.843397 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:11.843798 21594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:11.844302 21594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:11.844522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21519
I20260925 12:16:11.844640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:11.844872 392 external_mini_cluster.cc:1428] Setting key a71561f7e8366584a22c8ba3d6371f89
I20260925 12:16:11.846038 21594 catalog_manager.cc:1347] Generated new cluster ID: e9960d786c3c4531843adf11add87407
I20260925 12:16:11.846128 21594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:11.847193 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:11.845359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:53983 (local address 127.0.0.1:40641)
0925 12:16:11.845540 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:11.845549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:11.845587 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:16:11.845760 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:11.845787 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:11.845820 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:11.845966 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:11.846774 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:11.846778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:11.846779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:11.846783 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:11.846829 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:11.846840 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:11.846922 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:11.846988 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 12:16:11.847040 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260925 12:16:11.848042 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:11.857893 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:11.849482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:45285 (local address 127.0.0.1:40641)
0925 12:16:11.849519 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:11.849521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:11.856761 (+ 7240us) server_negotiation.cc:407] Connection header received
0925 12:16:11.856853 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:11.856854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:11.856861 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:11.856951 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:11.857642 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:11.857644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:11.857644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:11.857649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:11.857664 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:11.857669 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:11.857723 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:11.857782 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 12:16:11.857799 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:16:11.871114 21594 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:11.871332 21594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:11.874924 21594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045264dd49b4442daf203c1749fb50fe: Generated new TSK 0
I20260925 12:16:11.875178 21594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:11.878981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21519
2026-09-25T12:16:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:11 dist-test-slave-2hqq krb5kdc[21618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:11 dist-test-slave-2hqq krb5kdc[21618](info): set up 2 sockets
Sep 25 12:16:11 dist-test-slave-2hqq krb5kdc[21618](info): commencing operation
krb5kdc: starting...
W20260925 12:16:12.908756 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:12 dist-test-slave-2hqq krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:13.031045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:13.113530 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:13.113803 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:13.113845 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:13.115033 21631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:13.115108 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:13.115134 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:13.115166 21631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:13.115190 21631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:13.116766 21631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:13.117056 21631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:13.117298 21631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:13.122200 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:13.122191 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:13.122191 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:13.122340 21631 server_base.cc:1034] running on GCE node
I20260925 12:16:13.122702 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:13.123005 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:13.124205 21631 hybrid_clock.cc:648] HybridClock initialized: now 1790338573124178 us; error 34 us; skew 500 ppm
Sep 25 12:16:13 dist-test-slave-2hqq krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:13.126874 21631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:13.127411 21631 webserver.cc:460] Webserver started at http://127.0.0.1:42781/ using document root <none> and password file <none>
I20260925 12:16:13.127660 21631 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:13.127802 21631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:13.128002 21631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:13.129228 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d"
format_stamp: "Formatted at 2026-09-25 12:16:13 on dist-test-slave-2hqq"
server_key: "b6cd51ad01e74efcd36c0ab11307ffff"
server_key_iv: "f948d400a616976653096595fe868a43"
server_key_version: "encryptionkey@0"
I20260925 12:16:13.129597 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d"
format_stamp: "Formatted at 2026-09-25 12:16:13 on dist-test-slave-2hqq"
server_key: "b6cd51ad01e74efcd36c0ab11307ffff"
server_key_iv: "f948d400a616976653096595fe868a43"
server_key_version: "encryptionkey@0"
I20260925 12:16:13.130975 21631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 12:16:13.131815 21646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:13.132015 21631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:13.132125 21631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d"
format_stamp: "Formatted at 2026-09-25 12:16:13 on dist-test-slave-2hqq"
server_key: "b6cd51ad01e74efcd36c0ab11307ffff"
server_key_iv: "f948d400a616976653096595fe868a43"
server_key_version: "encryptionkey@0"
I20260925 12:16:13.132205 21631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:13.147267 21631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:13.149485 21631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:13.149684 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:13.153842 21631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38453
I20260925 12:16:13.153885 21698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38453 every 8 connection(s)
I20260925 12:16:13.154268 21631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:13.155308 21699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:13.156086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21631
I20260925 12:16:13.156208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:13.156399 392 external_mini_cluster.cc:1428] Setting key 9ce77b872bcd64d6f946209b392dd5d5
I20260925 12:16:13.158689 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d: Bootstrap starting.
I20260925 12:16:13.159279 21699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:13.159583 21699 log.cc:824] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:13.160324 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d: No bootstrap required, opened a new log
I20260925 12:16:13.162865 21699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260925 12:16:13.163021 21699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:13.163045 21699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbab1c3ca24b04b14d1b0c33985d7d, State: Initialized, Role: FOLLOWER
I20260925 12:16:13.163177 21699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260925 12:16:13.163244 21699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:13.163265 21699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:13.163331 21699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 0 FOLLOWER]: Advancing to term 1
Sep 25 12:16:13 dist-test-slave-2hqq krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 12:16:13.163930 21699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260925 12:16:13.164050 21699 leader_election.cc:302] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ccbab1c3ca24b04b14d1b0c33985d7d; no voters:
I20260925 12:16:13.164235 21699 leader_election.cc:288] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:13.164279 21704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:13.164395 21704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [term 1 LEADER]: Becoming Leader. State: Replica: 6ccbab1c3ca24b04b14d1b0c33985d7d, State: Running, Role: LEADER
I20260925 12:16:13.164556 21699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:13.164552 21704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260925 12:16:13.165891 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } } }
I20260925 12:16:13.166078 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:13.166468 21720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:13.166031 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccbab1c3ca24b04b14d1b0c33985d7d. Latest consensus state: current_term: 1 leader_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccbab1c3ca24b04b14d1b0c33985d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } } }
I20260925 12:16:13.166535 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:13.166985 21720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:13.167882 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:13.156985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:60099 (local address 127.0.0.1:38453)
0925 12:16:13.157242 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:13.157250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:13.157276 (+ 26us) server_negotiation.cc:407] Connection header received
0925 12:16:13.157669 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:13.157675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:13.157726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:13.157843 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:13.158653 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:13.159688 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:13.160700 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:13.161113 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:13.161195 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:13.164033 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:13.164039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:13.164039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:13.164051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:13.166183 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:13.166673 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:13.166677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:13.166678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:13.166779 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:13.167038 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:13.167039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:13.167039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:13.167279 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:13.167460 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:13.167575 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 12:16:13.167678 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260925 12:16:13.168919 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:13.170303 21720 catalog_manager.cc:1347] Generated new cluster ID: b9b1daa8ccc1439f89ae93ab6d2eee25
I20260925 12:16:13.170361 21720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:13.183154 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:13.173016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:45213 (local address 127.0.0.1:38453)
0925 12:16:13.173055 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:13.173057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:13.176778 (+ 3721us) server_negotiation.cc:407] Connection header received
0925 12:16:13.176863 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:13.176864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:13.176876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:13.176945 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:13.177550 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:13.178050 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:13.178811 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:13.179037 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:13.179126 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:13.180397 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:13.180399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:13.180399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:13.180407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:13.181949 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:13.182328 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:13.182329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:13.182330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:13.182392 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:13.182615 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:13.182616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:13.182617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:13.182733 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:13.182843 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:13.182968 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 12:16:13.183035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:16:13.193321 21720 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:13.193922 21720 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:13.199877 21720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ccbab1c3ca24b04b14d1b0c33985d7d: Generated new TSK 0
I20260925 12:16:13.200101 21720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:13.204067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21631
W20260925 12:16:13.208535 21616 connection.cc:535] client connection to 127.0.0.1:38453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38453): Connection reset by peer (error 104)
2026-09-25T12:16:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:13 dist-test-slave-2hqq krb5kdc[21731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:13 dist-test-slave-2hqq krb5kdc[21731](info): set up 2 sockets
Sep 25 12:16:13 dist-test-slave-2hqq krb5kdc[21731](info): commencing operation
krb5kdc: starting...
W20260925 12:16:14.244761 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:14 dist-test-slave-2hqq krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:14.364406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:14.447700 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:14.447980 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:14.448022 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:14.449291 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:14.449371 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:14.449396 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:14.449419 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:14.449447 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:14.451088 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:14.451406 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:14.451679 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:16:14.457134 21744 server_base.cc:1034] running on GCE node
W20260925 12:16:14.457094 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:14.457101 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:14.457101 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:14.457607 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:14.457947 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:14.459143 21744 hybrid_clock.cc:648] HybridClock initialized: now 1790338574459130 us; error 57 us; skew 500 ppm
Sep 25 12:16:14 dist-test-slave-2hqq krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:14.461972 21744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:14.462486 21744 webserver.cc:460] Webserver started at http://127.0.0.1:40763/ using document root <none> and password file <none>
I20260925 12:16:14.462745 21744 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:14.462822 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:14.462980 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:14.464078 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "777911a724d149d19b886428aa32e771"
format_stamp: "Formatted at 2026-09-25 12:16:14 on dist-test-slave-2hqq"
server_key: "02e7e2a9e37d24d3049f1001e7301d56"
server_key_iv: "46cbafd5bb739f02cbdb981ec84c0abd"
server_key_version: "encryptionkey@0"
I20260925 12:16:14.464457 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "777911a724d149d19b886428aa32e771"
format_stamp: "Formatted at 2026-09-25 12:16:14 on dist-test-slave-2hqq"
server_key: "02e7e2a9e37d24d3049f1001e7301d56"
server_key_iv: "46cbafd5bb739f02cbdb981ec84c0abd"
server_key_version: "encryptionkey@0"
I20260925 12:16:14.465912 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:16:14.466747 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:14.466954 21744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:14.467080 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "777911a724d149d19b886428aa32e771"
format_stamp: "Formatted at 2026-09-25 12:16:14 on dist-test-slave-2hqq"
server_key: "02e7e2a9e37d24d3049f1001e7301d56"
server_key_iv: "46cbafd5bb739f02cbdb981ec84c0abd"
server_key_version: "encryptionkey@0"
I20260925 12:16:14.467163 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:14.493646 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:14.496158 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:14.496338 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:14.500413 21744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34047
I20260925 12:16:14.500468 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34047 every 8 connection(s)
I20260925 12:16:14.500896 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:14.501931 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:14.504407 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771: Bootstrap starting.
I20260925 12:16:14.505061 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:14.505373 21812 log.cc:824] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:14.506122 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771: No bootstrap required, opened a new log
I20260925 12:16:14.507802 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260925 12:16:14.507943 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:14.507966 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777911a724d149d19b886428aa32e771, State: Initialized, Role: FOLLOWER
I20260925 12:16:14.508132 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260925 12:16:14.508266 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:14.508312 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:14.508369 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:14.508987 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260925 12:16:14.509109 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 777911a724d149d19b886428aa32e771; no voters:
I20260925 12:16:14.509310 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:14.509352 21815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:14.509527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21744
I20260925 12:16:14.509559 21815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [term 1 LEADER]: Becoming Leader. State: Replica: 777911a724d149d19b886428aa32e771, State: Running, Role: LEADER
I20260925 12:16:14.509611 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:14.509634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:14.509639 21815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260925 12:16:14.509860 392 external_mini_cluster.cc:1428] Setting key 28cdc883c9570ef92eb53a2bcd1a377c
I20260925 12:16:14.510128 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "777911a724d149d19b886428aa32e771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260925 12:16:14.510202 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:14.510571 21825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:14.510911 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 777911a724d149d19b886428aa32e771. Latest consensus state: current_term: 1 leader_uuid: "777911a724d149d19b886428aa32e771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777911a724d149d19b886428aa32e771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260925 12:16:14.510984 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:14.511138 21825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:14.513113 21825 catalog_manager.cc:1347] Generated new cluster ID: 5428de676e5748d78762d875e3f7acd7
I20260925 12:16:14.513178 21825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:14.516335 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:14.510575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:44145 (local address 127.0.0.1:34047)
0925 12:16:14.511121 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:14.511130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:14.511168 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:16:14.511305 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:14.511314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:14.511380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:14.511518 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:14.511776 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:14.513002 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:14.514142 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:14.514768 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:14.514884 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:14.515634 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:14.515638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:14.515638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:14.515845 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:14.515881 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:14.515895 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:14.515971 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:14.516094 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 12:16:14.516121 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":70,"threads_started":1}
I20260925 12:16:14.517498 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:14.522022 21825 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:14.522491 21825 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:14.524369 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:14.520889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:55203 (local address 127.0.0.1:34047)
0925 12:16:14.520945 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:14.520947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:14.520957 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:16:14.520977 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:14.520977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:14.520989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:14.521053 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:14.521177 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:14.521949 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:14.522988 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:14.523281 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:14.523347 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:14.524078 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:14.524079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:14.524080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:14.524084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:14.524098 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:14.524103 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:14.524158 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:14.524274 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 12:16:14.524286 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 12:16:14.532567 21825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 777911a724d149d19b886428aa32e771: Generated new TSK 0
I20260925 12:16:14.532814 21825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:14.536520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21744
W20260925 12:16:14.541406 21729 connection.cc:535] client connection to 127.0.0.1:34047 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34047): Connection reset by peer (error 104)
2026-09-25T12:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:14 dist-test-slave-2hqq krb5kdc[21843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:14 dist-test-slave-2hqq krb5kdc[21843](info): set up 2 sockets
Sep 25 12:16:14 dist-test-slave-2hqq krb5kdc[21843](info): commencing operation
krb5kdc: starting...
W20260925 12:16:15.604786 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:15 dist-test-slave-2hqq krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 12:16:15.724817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:15.806689 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:15.806895 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:15.806919 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:15.808079 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:15.808122 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:15.808136 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:15.808149 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:15.808202 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:15.809779 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:15.810021 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:15.810250 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:15.815176 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:15.815176 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:15.815296 21856 server_base.cc:1034] running on GCE node
W20260925 12:16:15.815176 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:15.815683 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:15.815959 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:15.817116 21856 hybrid_clock.cc:648] HybridClock initialized: now 1790338575817099 us; error 27 us; skew 500 ppm
Sep 25 12:16:15 dist-test-slave-2hqq krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:15.819726 21856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 12:16:15.820250 21856 webserver.cc:460] Webserver started at http://127.0.0.1:39047/ using document root <none> and password file <none>
I20260925 12:16:15.820479 21856 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:15.820590 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:15.820748 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:15.821878 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "ca5710c270564a4780bbdeb9d3512e9b"
format_stamp: "Formatted at 2026-09-25 12:16:15 on dist-test-slave-2hqq"
server_key: "c8c5d81af3e76511a2adab924110d690"
server_key_iv: "ee35001c89887f247936ef8d51939f50"
server_key_version: "encryptionkey@0"
I20260925 12:16:15.822238 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca5710c270564a4780bbdeb9d3512e9b"
format_stamp: "Formatted at 2026-09-25 12:16:15 on dist-test-slave-2hqq"
server_key: "c8c5d81af3e76511a2adab924110d690"
server_key_iv: "ee35001c89887f247936ef8d51939f50"
server_key_version: "encryptionkey@0"
I20260925 12:16:15.823626 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:16:15.824422 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:15.824637 21856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:16:15.824738 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "ca5710c270564a4780bbdeb9d3512e9b"
format_stamp: "Formatted at 2026-09-25 12:16:15 on dist-test-slave-2hqq"
server_key: "c8c5d81af3e76511a2adab924110d690"
server_key_iv: "ee35001c89887f247936ef8d51939f50"
server_key_version: "encryptionkey@0"
I20260925 12:16:15.824829 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:15.828145 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:15.830138 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:15.830303 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:15.834342 21856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39391
I20260925 12:16:15.834369 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39391 every 8 connection(s)
I20260925 12:16:15.835083 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:15.836143 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:15.838536 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b: Bootstrap starting.
I20260925 12:16:15.839078 21924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:15.839326 21924 log.cc:824] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:15.839933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21856
I20260925 12:16:15.840060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:15.840134 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b: No bootstrap required, opened a new log
I20260925 12:16:15.840299 392 external_mini_cluster.cc:1428] Setting key e2eff230d9cd4f3b888781b86b3afcba
I20260925 12:16:15.842133 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 12:16:15.842311 21924 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:15.842355 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5710c270564a4780bbdeb9d3512e9b, State: Initialized, Role: FOLLOWER
I20260925 12:16:15.842550 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 12:16:15.842669 21924 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:15.842728 21924 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:15.842782 21924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:15.843501 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:15.840870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:60817 (local address 127.0.0.1:39391)
0925 12:16:15.841046 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:15.841056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:15.841091 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:16:15.842115 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:15.842121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:15.842146 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:15.842263 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:15.843026 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:15.843030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:15.843030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:15.843034 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:15.843125 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:15.843138 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:15.843214 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:15.843285 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 12:16:15.843328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260925 12:16:15.843899 21924 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 12:16:15.844019 21924 leader_election.cc:302] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5710c270564a4780bbdeb9d3512e9b; no voters:
I20260925 12:16:15.844206 21924 leader_election.cc:288] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:15.844229 21929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:15.844340 21929 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [term 1 LEADER]: Becoming Leader. State: Replica: ca5710c270564a4780bbdeb9d3512e9b, State: Running, Role: LEADER
I20260925 12:16:15.844381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:15.844427 21929 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 12:16:15.844476 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:15.844784 21929 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5710c270564a4780bbdeb9d3512e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260925 12:16:15.844909 21929 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:15.845103 21929 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5710c270564a4780bbdeb9d3512e9b. Latest consensus state: current_term: 1 leader_uuid: "ca5710c270564a4780bbdeb9d3512e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5710c270564a4780bbdeb9d3512e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260925 12:16:15.845227 21929 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:15.845571 21936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:15.846084 21936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:15.847625 21936 catalog_manager.cc:1347] Generated new cluster ID: 9ab0dae54f79481ca21b52516846b598
I20260925 12:16:15.847677 21936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:15.849727 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:15.846047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:35089 (local address 127.0.0.1:39391)
0925 12:16:15.846091 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:15.846093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:15.848686 (+ 2593us) server_negotiation.cc:407] Connection header received
0925 12:16:15.848777 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:15.848778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:15.848784 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:15.848811 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:15.849515 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:15.849516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:15.849516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:15.849520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:15.849533 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:15.849538 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:15.849563 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:15.849653 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 12:16:15.849666 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 12:16:15.866712 21936 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:15.866987 21936 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:15.873842 21936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca5710c270564a4780bbdeb9d3512e9b: Generated new TSK 0
I20260925 12:16:15.874085 21936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:15.881659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21856
2026-09-25T12:16:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:15 dist-test-slave-2hqq krb5kdc[21955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:15 dist-test-slave-2hqq krb5kdc[21955](info): set up 2 sockets
Sep 25 12:16:15 dist-test-slave-2hqq krb5kdc[21955](info): commencing operation
krb5kdc: starting...
W20260925 12:16:16.912892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:16 dist-test-slave-2hqq krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:17.031504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:17.119030 21968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:17.119288 21968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:17.119326 21968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:17.120558 21968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:17.120640 21968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:17.120663 21968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:17.120684 21968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:17.120713 21968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:17.122267 21968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:17.122568 21968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:17.122812 21968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:17.127758 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:17.127847 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:17.127832 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:17.128019 21968 server_base.cc:1034] running on GCE node
I20260925 12:16:17.128275 21968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:17.128634 21968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:17.129822 21968 hybrid_clock.cc:648] HybridClock initialized: now 1790338577129806 us; error 44 us; skew 500 ppm
Sep 25 12:16:17 dist-test-slave-2hqq krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:17.140115 21968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:17.140770 21968 webserver.cc:460] Webserver started at http://127.0.98.62:35795/ using document root <none> and password file <none>
I20260925 12:16:17.140997 21968 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:17.141047 21968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:17.141145 21968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:17.142233 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "f214922366cc40d48f0b2135b45d14a5"
format_stamp: "Formatted at 2026-09-25 12:16:17 on dist-test-slave-2hqq"
server_key: "4f320176ec7d1dd0367104644d0f6d64"
server_key_iv: "f491729383ab8f3988add699907fa5fb"
server_key_version: "encryptionkey@0"
I20260925 12:16:17.142549 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "f214922366cc40d48f0b2135b45d14a5"
format_stamp: "Formatted at 2026-09-25 12:16:17 on dist-test-slave-2hqq"
server_key: "4f320176ec7d1dd0367104644d0f6d64"
server_key_iv: "f491729383ab8f3988add699907fa5fb"
server_key_version: "encryptionkey@0"
I20260925 12:16:17.143959 21968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:17.144802 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:17.144976 21968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:17.145066 21968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "f214922366cc40d48f0b2135b45d14a5"
format_stamp: "Formatted at 2026-09-25 12:16:17 on dist-test-slave-2hqq"
server_key: "4f320176ec7d1dd0367104644d0f6d64"
server_key_iv: "f491729383ab8f3988add699907fa5fb"
server_key_version: "encryptionkey@0"
I20260925 12:16:17.145129 21968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:17.170782 21968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:17.172762 21968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:17.172878 21968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:17.176869 21968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45565
I20260925 12:16:17.176947 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45565 every 8 connection(s)
I20260925 12:16:17.177311 21968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:17.178231 22036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:17.180586 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5: Bootstrap starting.
I20260925 12:16:17.181182 22036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:17.181470 22036 log.cc:824] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:17.182184 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5: No bootstrap required, opened a new log
I20260925 12:16:17.183817 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260925 12:16:17.183959 22036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:17.183982 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f214922366cc40d48f0b2135b45d14a5, State: Initialized, Role: FOLLOWER
I20260925 12:16:17.184165 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260925 12:16:17.184226 22036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:17.184247 22036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:17.184314 22036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:17.184928 22036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260925 12:16:17.185042 22036 leader_election.cc:302] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f214922366cc40d48f0b2135b45d14a5; no voters:
I20260925 12:16:17.185240 22036 leader_election.cc:288] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:17.185257 22039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:17.185429 22039 raft_consensus.cc:686] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [term 1 LEADER]: Becoming Leader. State: Replica: f214922366cc40d48f0b2135b45d14a5, State: Running, Role: LEADER
I20260925 12:16:17.185504 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:17.185560 22039 consensus_queue.cc:237] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } }
I20260925 12:16:17.185988 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f214922366cc40d48f0b2135b45d14a5. Latest consensus state: current_term: 1 leader_uuid: "f214922366cc40d48f0b2135b45d14a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260925 12:16:17.186125 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f214922366cc40d48f0b2135b45d14a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f214922366cc40d48f0b2135b45d14a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45565 } } }
I20260925 12:16:17.186132 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:17.186224 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:17.186591 22048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:17.187060 22048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:17.187188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 21968
I20260925 12:16:17.187296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:17.187496 392 external_mini_cluster.cc:1428] Setting key 65182b5cc65737fa1c5b2e4e6725474e
I20260925 12:16:17.189515 22048 catalog_manager.cc:1347] Generated new cluster ID: af2aeab21ce149988a348c4cc7203e51
I20260925 12:16:17.189574 22048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:16:17 dist-test-slave-2hqq krb5kdc[21955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:17.197866 22048 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:17.198348 22048 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:17.199803 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:17.187985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41441 (local address 127.0.98.62:45565)
0925 12:16:17.188153 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:17.188163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:17.188387 (+ 224us) server_negotiation.cc:407] Connection header received
0925 12:16:17.188607 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:17.188615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:17.188662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:17.188806 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:17.189482 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:17.190908 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:17.192037 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:17.192656 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:17.192762 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:17.195781 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:17.195785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:17.195786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:17.195799 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:17.197881 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:17.198368 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:17.198372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:17.198372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:17.198486 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:17.198721 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:17.198721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:17.198722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:17.198993 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:17.199180 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:17.199460 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 12:16:17.199563 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260925 12:16:17.201004 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:17.205199 22048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f214922366cc40d48f0b2135b45d14a5: Generated new TSK 0
I20260925 12:16:17.205376 22048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:17.209235 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:17.202497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35831 (local address 127.0.98.62:45565)
0925 12:16:17.202530 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:17.202532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:17.202628 (+ 96us) server_negotiation.cc:407] Connection header received
0925 12:16:17.202671 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:17.202672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:17.202682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:17.202736 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:17.203282 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:17.204074 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:17.205123 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:17.205446 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:17.205515 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:17.206640 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:17.206641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:17.206642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:17.206649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:17.208121 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:17.208472 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:17.208473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:17.208474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:17.208547 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:17.208742 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:17.208743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:17.208743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:17.208870 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:17.208998 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:17.209087 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 12:16:17.209149 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:16:17.211086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 21968
W20260925 12:16:17.215523 21953 connection.cc:535] client connection to 127.0.98.62:45565 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45565): Connection reset by peer (error 104)
2026-09-25T12:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:17 dist-test-slave-2hqq krb5kdc[22067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:17 dist-test-slave-2hqq krb5kdc[22067](info): set up 2 sockets
Sep 25 12:16:17 dist-test-slave-2hqq krb5kdc[22067](info): commencing operation
krb5kdc: starting...
W20260925 12:16:18.280922 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:18 dist-test-slave-2hqq krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:18.403564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:18.484440 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:18.484676 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:18.484700 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:18.485809 22080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:18.485848 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:18.485862 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:18.485875 22080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:18.485888 22080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:18.487288 22080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:18.487480 22080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:18.487646 22080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:16:18.493177 22080 server_base.cc:1034] running on GCE node
W20260925 12:16:18.493337 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:18.493403 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:18.493171 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:18.493688 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:18.493976 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:18.495141 22080 hybrid_clock.cc:648] HybridClock initialized: now 1790338578495140 us; error 35 us; skew 500 ppm
Sep 25 12:16:18 dist-test-slave-2hqq krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:18.497692 22080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:18.498095 22080 webserver.cc:460] Webserver started at http://127.0.98.62:37881/ using document root <none> and password file <none>
I20260925 12:16:18.498291 22080 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:18.498337 22080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:18.498425 22080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:18.499445 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19"
format_stamp: "Formatted at 2026-09-25 12:16:18 on dist-test-slave-2hqq"
server_key: "ce8f2c16fd237d4a53134ffa5c9408e0"
server_key_iv: "866cd1fa4bf3d6e1988ae2754a634e10"
server_key_version: "encryptionkey@0"
I20260925 12:16:18.499745 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19"
format_stamp: "Formatted at 2026-09-25 12:16:18 on dist-test-slave-2hqq"
server_key: "ce8f2c16fd237d4a53134ffa5c9408e0"
server_key_iv: "866cd1fa4bf3d6e1988ae2754a634e10"
server_key_version: "encryptionkey@0"
I20260925 12:16:18.501092 22080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:18.501816 22095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:18.502014 22080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:16:18.502100 22080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19"
format_stamp: "Formatted at 2026-09-25 12:16:18 on dist-test-slave-2hqq"
server_key: "ce8f2c16fd237d4a53134ffa5c9408e0"
server_key_iv: "866cd1fa4bf3d6e1988ae2754a634e10"
server_key_version: "encryptionkey@0"
I20260925 12:16:18.502156 22080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:18.517184 22080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:18.519567 22080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:18.519680 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:18.523597 22080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38483
I20260925 12:16:18.523931 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38483 every 8 connection(s)
I20260925 12:16:18.523975 22080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:18.524746 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:18.526983 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19: Bootstrap starting.
I20260925 12:16:18.527575 22148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:18.527864 22148 log.cc:824] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:18.528266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22080
I20260925 12:16:18.528405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:18.528636 392 external_mini_cluster.cc:1428] Setting key e4a5063cd7095760793965d076be22ca
I20260925 12:16:18.528697 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19: No bootstrap required, opened a new log
I20260925 12:16:18.531659 22148 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260925 12:16:18.531836 22148 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:18.531867 22148 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a9f567d5f64ce3992c8dec9b4b9a19, State: Initialized, Role: FOLLOWER
I20260925 12:16:18.531982 22148 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260925 12:16:18.532063 22148 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:18.532102 22148 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:18.532145 22148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:18.532861 22148 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260925 12:16:18.532979 22148 leader_election.cc:302] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a9f567d5f64ce3992c8dec9b4b9a19; no voters:
I20260925 12:16:18.533151 22148 leader_election.cc:288] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:18.533226 22153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:18.533396 22148 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:18.533411 22153 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [term 1 LEADER]: Becoming Leader. State: Replica: a0a9f567d5f64ce3992c8dec9b4b9a19, State: Running, Role: LEADER
I20260925 12:16:18.533615 22153 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260925 12:16:18.533990 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } } }
I20260925 12:16:18.533995 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a9f567d5f64ce3992c8dec9b4b9a19. Latest consensus state: current_term: 1 leader_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a9f567d5f64ce3992c8dec9b4b9a19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } } }
I20260925 12:16:18.534094 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:18.534097 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:18.534313 22162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:18.534883 22162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:18.536291 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:18.529169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54403 (local address 127.0.98.62:38483)
0925 12:16:18.529371 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:18.529382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:18.529421 (+ 39us) server_negotiation.cc:407] Connection header received
0925 12:16:18.530111 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:18.530120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:18.530185 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:18.530367 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:18.530552 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:18.531996 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:18.532984 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:18.533623 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:18.534126 (+ 503us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:18.535475 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:18.535479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:18.535480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:18.535689 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:18.535729 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:18.535744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:18.535832 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:18.536037 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 12:16:18.536064 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260925 12:16:18.536940 22162 catalog_manager.cc:1347] Generated new cluster ID: 878e37dc10e34551ab98d70e19a35037
I20260925 12:16:18.536993 22162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:18.537343 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:18.544577 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:18.538778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54531 (local address 127.0.98.62:38483)
0925 12:16:18.538812 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:18.538814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:18.540692 (+ 1878us) server_negotiation.cc:407] Connection header received
0925 12:16:18.541080 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:18.541081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:18.541092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:18.541154 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:18.541360 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:18.542094 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:18.543160 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:18.543477 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:18.543536 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:18.544304 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:18.544305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:18.544305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:18.544310 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:18.544324 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:18.544328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:18.544383 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:18.544464 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 12:16:18.544475 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:16:18.548775 22162 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:18.549302 22162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:18.559819 22162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a9f567d5f64ce3992c8dec9b4b9a19: Generated new TSK 0
I20260925 12:16:18.560065 22162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:18.566551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22080
W20260925 12:16:18.571059 22065 connection.cc:535] client connection to 127.0.98.62:38483 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38483): Connection reset by peer (error 104)
2026-09-25T12:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:18 dist-test-slave-2hqq krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:18 dist-test-slave-2hqq krb5kdc[22179](info): set up 2 sockets
Sep 25 12:16:18 dist-test-slave-2hqq krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260925 12:16:19.604647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:19 dist-test-slave-2hqq krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:19.727737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:19.809604 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:19.809871 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:19.809908 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:19.811033 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:19.811103 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:19.811126 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:19.811147 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:19.811175 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:19.812713 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:19.813007 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:19.813233 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:19.818133 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:19.818120 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:19.818120 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:19.818336 22192 server_base.cc:1034] running on GCE node
I20260925 12:16:19.818595 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:19.818897 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:19.820068 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790338579820055 us; error 46 us; skew 500 ppm
Sep 25 12:16:19 dist-test-slave-2hqq krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:19.822640 22192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:19.823145 22192 webserver.cc:460] Webserver started at http://127.0.98.62:38119/ using document root <none> and password file <none>
I20260925 12:16:19.823376 22192 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:19.823451 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:19.823603 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:19.824800 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "e9846523fd4c4e3db41d9f021158789b"
format_stamp: "Formatted at 2026-09-25 12:16:19 on dist-test-slave-2hqq"
server_key: "2255ce893faeabf2878bd6a48bd0111f"
server_key_iv: "ccc4a781f437a273ccb36b35f0e7d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:16:19.825165 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9846523fd4c4e3db41d9f021158789b"
format_stamp: "Formatted at 2026-09-25 12:16:19 on dist-test-slave-2hqq"
server_key: "2255ce893faeabf2878bd6a48bd0111f"
server_key_iv: "ccc4a781f437a273ccb36b35f0e7d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:16:19.826642 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:16:19.827469 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:19.827653 22192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:19.827775 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e9846523fd4c4e3db41d9f021158789b"
format_stamp: "Formatted at 2026-09-25 12:16:19 on dist-test-slave-2hqq"
server_key: "2255ce893faeabf2878bd6a48bd0111f"
server_key_iv: "ccc4a781f437a273ccb36b35f0e7d7d3"
server_key_version: "encryptionkey@0"
I20260925 12:16:19.827853 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:19.831321 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:19.833287 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:19.833446 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:19.837486 22192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41935
I20260925 12:16:19.837517 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41935 every 8 connection(s)
I20260925 12:16:19.837828 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:19.838738 22260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:19.841065 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: Bootstrap starting.
I20260925 12:16:19.841630 22260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:19.841910 22260 log.cc:824] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:19.842571 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: No bootstrap required, opened a new log
I20260925 12:16:19.842625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22192
I20260925 12:16:19.842746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:19.842944 392 external_mini_cluster.cc:1428] Setting key 087fe4a3158481d8ada1fc8ea1fa3b35
I20260925 12:16:19.844895 22260 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260925 12:16:19.845099 22260 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:19.845132 22260 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9846523fd4c4e3db41d9f021158789b, State: Initialized, Role: FOLLOWER
I20260925 12:16:19.845260 22260 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260925 12:16:19.845340 22260 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:19.845427 22260 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:19.845487 22260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:19.846042 22260 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260925 12:16:19.846150 22260 leader_election.cc:302] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9846523fd4c4e3db41d9f021158789b; no voters:
I20260925 12:16:19.846325 22260 leader_election.cc:288] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:19.846365 22265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:19.846589 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:19.843481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36073 (local address 127.0.98.62:41935)
0925 12:16:19.843669 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:19.843679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:19.843711 (+ 32us) server_negotiation.cc:407] Connection header received
0925 12:16:19.845099 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:19.845108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:19.845146 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:19.845294 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:19.846045 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:19.846049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:19.846050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:19.846055 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:19.846102 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:19.846118 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:19.846190 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:19.846334 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 12:16:19.846381 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260925 12:16:19.846607 22265 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [term 1 LEADER]: Becoming Leader. State: Replica: e9846523fd4c4e3db41d9f021158789b, State: Running, Role: LEADER
I20260925 12:16:19.846664 22260 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:19.847043 22265 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260925 12:16:19.847571 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:19.848647 22266 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9846523fd4c4e3db41d9f021158789b. Latest consensus state: current_term: 1 leader_uuid: "e9846523fd4c4e3db41d9f021158789b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260925 12:16:19.849205 22266 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:19.849290 22280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:19.849334 22280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:19.848791 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9846523fd4c4e3db41d9f021158789b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9846523fd4c4e3db41d9f021158789b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260925 12:16:19.849455 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:19.849650 22287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:19.849879 22287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:19.850332 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:19.849127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51137 (local address 127.0.98.62:41935)
0925 12:16:19.849165 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:19.849167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:19.849275 (+ 108us) server_negotiation.cc:407] Connection header received
0925 12:16:19.849382 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:19.849383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:19.849388 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:19.849415 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:19.850114 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:19.850115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:19.850115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:19.850119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:19.850135 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:19.850139 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:19.850202 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:19.850266 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 12:16:19.850279 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:16:19.851410 22287 catalog_manager.cc:1347] Generated new cluster ID: 1b9d34fae0ad4b01889f950a5a8c369f
I20260925 12:16:19.851467 22287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:19.865875 22287 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:19.866091 22287 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:19.870046 22287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9846523fd4c4e3db41d9f021158789b: Generated new TSK 0
I20260925 12:16:19.870294 22287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:19.875801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22192
2026-09-25T12:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:19 dist-test-slave-2hqq krb5kdc[22291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:19 dist-test-slave-2hqq krb5kdc[22291](info): set up 2 sockets
Sep 25 12:16:19 dist-test-slave-2hqq krb5kdc[22291](info): commencing operation
krb5kdc: starting...
W20260925 12:16:20.928793 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:21.047549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:21.129032 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:21.129244 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:21.129266 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:21.130471 22304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:21.130517 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:21.130534 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:21.130548 22304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:21.130592 22304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:21.132186 22304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:21.132436 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:21.132689 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:21.137455 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:21.137468 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:21.137478 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:21.137665 22304 server_base.cc:1034] running on GCE node
I20260925 12:16:21.137969 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:21.138230 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:21.139403 22304 hybrid_clock.cc:648] HybridClock initialized: now 1790338581139394 us; error 35 us; skew 500 ppm
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:21.142107 22304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:21.142599 22304 webserver.cc:460] Webserver started at http://127.0.98.62:34311/ using document root <none> and password file <none>
I20260925 12:16:21.142853 22304 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:21.142920 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:21.143059 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:21.144124 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "e307522ccd5a484bbe495f48614020d0"
format_stamp: "Formatted at 2026-09-25 12:16:21 on dist-test-slave-2hqq"
server_key: "f5ec5fd5f3a265b7749b0ddeea368341"
server_key_iv: "156b6e44f82bbdbe5a7d2aeba5a223ef"
server_key_version: "encryptionkey@0"
I20260925 12:16:21.144550 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "e307522ccd5a484bbe495f48614020d0"
format_stamp: "Formatted at 2026-09-25 12:16:21 on dist-test-slave-2hqq"
server_key: "f5ec5fd5f3a265b7749b0ddeea368341"
server_key_iv: "156b6e44f82bbdbe5a7d2aeba5a223ef"
server_key_version: "encryptionkey@0"
I20260925 12:16:21.146067 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:16:21.147033 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:21.147233 22304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 12:16:21.147341 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e307522ccd5a484bbe495f48614020d0"
format_stamp: "Formatted at 2026-09-25 12:16:21 on dist-test-slave-2hqq"
server_key: "f5ec5fd5f3a265b7749b0ddeea368341"
server_key_iv: "156b6e44f82bbdbe5a7d2aeba5a223ef"
server_key_version: "encryptionkey@0"
I20260925 12:16:21.147441 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:21.160933 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:21.163337 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:21.163509 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:21.167668 22304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35677
I20260925 12:16:21.167703 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35677 every 8 connection(s)
I20260925 12:16:21.168098 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:21.169024 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:21.171334 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0: Bootstrap starting.
I20260925 12:16:21.171908 22372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:21.172188 22372 log.cc:824] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:21.172541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22304
I20260925 12:16:21.172662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:21.172842 392 external_mini_cluster.cc:1428] Setting key dfc675ffd9884f9d5eb127f4c01ca96b
I20260925 12:16:21.172994 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0: No bootstrap required, opened a new log
I20260925 12:16:21.175248 22372 raft_consensus.cc:348] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } }
I20260925 12:16:21.175422 22372 raft_consensus.cc:374] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:21.175457 22372 raft_consensus.cc:729] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e307522ccd5a484bbe495f48614020d0, State: Initialized, Role: FOLLOWER
I20260925 12:16:21.175635 22372 consensus_queue.cc:260] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } }
I20260925 12:16:21.175724 22372 raft_consensus.cc:388] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:21.175765 22372 raft_consensus.cc:482] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:21.175808 22372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:21.176467 22372 raft_consensus.cc:504] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } }
I20260925 12:16:21.176602 22372 leader_election.cc:302] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e307522ccd5a484bbe495f48614020d0; no voters:
I20260925 12:16:21.176744 22372 leader_election.cc:288] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:21.176810 22377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:21.176965 22372 sys_catalog.cc:564] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:21.177019 22377 raft_consensus.cc:686] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [term 1 LEADER]: Becoming Leader. State: Replica: e307522ccd5a484bbe495f48614020d0, State: Running, Role: LEADER
I20260925 12:16:21.177143 22377 consensus_queue.cc:237] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } }
I20260925 12:16:21.177562 22378 sys_catalog.cc:455] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e307522ccd5a484bbe495f48614020d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } } }
I20260925 12:16:21.177637 22378 sys_catalog.cc:458] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:21.177807 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e307522ccd5a484bbe495f48614020d0. Latest consensus state: current_term: 1 leader_uuid: "e307522ccd5a484bbe495f48614020d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e307522ccd5a484bbe495f48614020d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35677 } } }
I20260925 12:16:21.177861 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:21.178283 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:21.179075 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:21.180517 22386 catalog_manager.cc:1347] Generated new cluster ID: 2efcd529227a44b5b0db78116ae995aa
I20260925 12:16:21.180588 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:21.186800 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:21.173333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59569 (local address 127.0.98.62:35677)
0925 12:16:21.173525 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:21.173534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:21.173569 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:16:21.174345 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:21.174354 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:21.174423 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:21.174576 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:21.175273 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:21.177033 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:21.178276 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:21.178850 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:21.178969 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:21.183117 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:21.183122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:21.183122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:21.183134 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:21.185088 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:21.185517 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:21.185520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:21.185521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:21.185645 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:21.185860 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:21.185861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:21.185861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:21.186154 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:21.186302 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:21.186437 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 12:16:21.186564 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 12:16:21.187740 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:21.189121 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:21.189828 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:21.195751 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:21.189365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47383 (local address 127.0.98.62:35677)
0925 12:16:21.189401 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:21.189403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:21.189514 (+ 111us) server_negotiation.cc:407] Connection header received
0925 12:16:21.189593 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:21.189594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:21.189838 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:21.189908 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:21.190449 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:21.190956 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:21.191687 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:21.191902 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:21.191984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:21.193241 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:21.193242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:21.193243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:21.193250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:21.194639 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:21.194983 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:21.194984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:21.194985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:21.195036 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:21.195243 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:21.195243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:21.195244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:21.195352 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:21.195456 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:21.195559 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 12:16:21.195648 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:16:21.197839 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e307522ccd5a484bbe495f48614020d0: Generated new TSK 0
I20260925 12:16:21.198033 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:21.200198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22304
2026-09-25T12:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22403](info): set up 2 sockets
Sep 25 12:16:21 dist-test-slave-2hqq krb5kdc[22403](info): commencing operation
krb5kdc: starting...
W20260925 12:16:22.236652 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:22 dist-test-slave-2hqq krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:22.356642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:22.439430 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:22.439692 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:22.439735 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:22.440917 22416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:22.440994 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:22.441030 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:22.441062 22416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:22.441087 22416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:22.442569 22416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:22.442853 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:22.443089 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:22.447957 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:22.447957 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:22.447999 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:22.448205 22416 server_base.cc:1034] running on GCE node
I20260925 12:16:22.448545 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:22.448799 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:22.449940 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790338582449930 us; error 28 us; skew 500 ppm
Sep 25 12:16:22 dist-test-slave-2hqq krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:22.455947 22416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:22.456416 22416 webserver.cc:460] Webserver started at http://127.0.98.62:36815/ using document root <none> and password file <none>
I20260925 12:16:22.456686 22416 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:22.456758 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:22.456905 22416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:22.457980 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "d0ab9d4cad584fcd9c47ea57b841c93c"
format_stamp: "Formatted at 2026-09-25 12:16:22 on dist-test-slave-2hqq"
server_key: "2d7fe77f40ff1c3e64b707a09339701f"
server_key_iv: "dcbba6a16bef7d5e5254610b29517268"
server_key_version: "encryptionkey@0"
I20260925 12:16:22.458348 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0ab9d4cad584fcd9c47ea57b841c93c"
format_stamp: "Formatted at 2026-09-25 12:16:22 on dist-test-slave-2hqq"
server_key: "2d7fe77f40ff1c3e64b707a09339701f"
server_key_iv: "dcbba6a16bef7d5e5254610b29517268"
server_key_version: "encryptionkey@0"
I20260925 12:16:22.459671 22416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:22.460465 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:22.460724 22416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:22.460817 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "d0ab9d4cad584fcd9c47ea57b841c93c"
format_stamp: "Formatted at 2026-09-25 12:16:22 on dist-test-slave-2hqq"
server_key: "2d7fe77f40ff1c3e64b707a09339701f"
server_key_iv: "dcbba6a16bef7d5e5254610b29517268"
server_key_version: "encryptionkey@0"
I20260925 12:16:22.460901 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:22.473253 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:22.475365 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:22.475530 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:22.479774 22416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38907
I20260925 12:16:22.479809 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38907 every 8 connection(s)
I20260925 12:16:22.480183 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:22.481139 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:22.481804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22416
I20260925 12:16:22.481938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:22.482136 392 external_mini_cluster.cc:1428] Setting key 0755cd556ad536144e9d2d8ab9135a35
I20260925 12:16:22.484390 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: Bootstrap starting.
I20260925 12:16:22.485117 22484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:22.485421 22484 log.cc:824] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:22.486195 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: No bootstrap required, opened a new log
I20260925 12:16:22.488222 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:22.482705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44909 (local address 127.0.98.62:38907)
0925 12:16:22.482901 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:22.482910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:22.482945 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:16:22.483520 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:22.483536 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:22.483602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:22.483756 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:22.483929 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:22.485313 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:22.486130 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:22.486740 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:22.486851 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:22.487514 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:22.487518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:22.487518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:22.487758 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:22.487796 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:22.487810 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:22.487880 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:22.487936 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 12:16:22.488001 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260925 12:16:22.488838 22484 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260925 12:16:22.489013 22484 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:22.489056 22484 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ab9d4cad584fcd9c47ea57b841c93c, State: Initialized, Role: FOLLOWER
I20260925 12:16:22.489187 22484 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260925 12:16:22.489269 22484 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:22.489316 22484 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:22.489365 22484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:22.489383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:22.489907 22484 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260925 12:16:22.489990 22484 leader_election.cc:302] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ab9d4cad584fcd9c47ea57b841c93c; no voters:
I20260925 12:16:22.490221 22484 leader_election.cc:288] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:22.490262 22489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:22.490429 22489 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [term 1 LEADER]: Becoming Leader. State: Replica: d0ab9d4cad584fcd9c47ea57b841c93c, State: Running, Role: LEADER
I20260925 12:16:22.490468 22484 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:22.491972 22489 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260925 12:16:22.492576 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ab9d4cad584fcd9c47ea57b841c93c. Latest consensus state: current_term: 1 leader_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } } }
I20260925 12:16:22.492799 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:22.492658 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ab9d4cad584fcd9c47ea57b841c93c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } } }
I20260925 12:16:22.493023 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:22.493387 22508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:22.493454 22508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:22.493515 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:22.493709 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:22.494400 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:22.490886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34585 (local address 127.0.98.62:38907)
0925 12:16:22.490920 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:22.490922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:22.491054 (+ 132us) server_negotiation.cc:407] Connection header received
0925 12:16:22.491119 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:22.491120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:22.491130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:22.491180 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:22.491359 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:22.492069 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:22.493164 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:22.493455 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:22.493524 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:22.494180 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:22.494182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:22.494182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:22.494186 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:22.494199 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:22.494202 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:22.494243 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:22.494306 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:16:22.494324 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 12:16:22.495361 22509 catalog_manager.cc:1347] Generated new cluster ID: c230e1a2975949b8b95a2327ca890808
I20260925 12:16:22.495430 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:22.513573 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:22.514146 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:22.518292 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ab9d4cad584fcd9c47ea57b841c93c: Generated new TSK 0
I20260925 12:16:22.518513 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:22.523653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22416
2026-09-25T12:16:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:22 dist-test-slave-2hqq krb5kdc[22515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:22 dist-test-slave-2hqq krb5kdc[22515](info): set up 2 sockets
Sep 25 12:16:22 dist-test-slave-2hqq krb5kdc[22515](info): commencing operation
krb5kdc: starting...
W20260925 12:16:23.552685 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:23 dist-test-slave-2hqq krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:23.671655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:23.752692 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:23.752904 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:23.752928 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:23.754082 22528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:23.754128 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:23.754143 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:23.754156 22528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:23.754168 22528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:23.755668 22528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:23.755927 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:23.756174 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:23.761404 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:23.761389 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:23.761389 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:23.761417 22528 server_base.cc:1034] running on GCE node
I20260925 12:16:23.761865 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:23.762112 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:23.763254 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790338583763241 us; error 29 us; skew 500 ppm
Sep 25 12:16:23 dist-test-slave-2hqq krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:23.765758 22528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:23.766227 22528 webserver.cc:460] Webserver started at http://127.0.98.62:44819/ using document root <none> and password file <none>
I20260925 12:16:23.766426 22528 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:23.766466 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:23.766553 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:23.767552 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "e5a52e44ccd14d0995aedbbd60a122b7"
format_stamp: "Formatted at 2026-09-25 12:16:23 on dist-test-slave-2hqq"
server_key: "6f5bf3c72ba29714495463ccf8255ce1"
server_key_iv: "995dac0dcf48371f79b2be0e6f3b4cc1"
server_key_version: "encryptionkey@0"
I20260925 12:16:23.767845 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5a52e44ccd14d0995aedbbd60a122b7"
format_stamp: "Formatted at 2026-09-25 12:16:23 on dist-test-slave-2hqq"
server_key: "6f5bf3c72ba29714495463ccf8255ce1"
server_key_iv: "995dac0dcf48371f79b2be0e6f3b4cc1"
server_key_version: "encryptionkey@0"
I20260925 12:16:23.769176 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:23.769870 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:23.770077 22528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:16:23.770161 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "e5a52e44ccd14d0995aedbbd60a122b7"
format_stamp: "Formatted at 2026-09-25 12:16:23 on dist-test-slave-2hqq"
server_key: "6f5bf3c72ba29714495463ccf8255ce1"
server_key_iv: "995dac0dcf48371f79b2be0e6f3b4cc1"
server_key_version: "encryptionkey@0"
I20260925 12:16:23.770215 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:23.773473 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:23.775729 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:23.775835 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:23.779887 22528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41829
I20260925 12:16:23.779958 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41829 every 8 connection(s)
I20260925 12:16:23.780287 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:23.781484 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:23.783715 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7: Bootstrap starting.
I20260925 12:16:23.784314 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:23.784653 22596 log.cc:824] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:23.785369 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7: No bootstrap required, opened a new log
I20260925 12:16:23.786358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22528
I20260925 12:16:23.786492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:23.786692 392 external_mini_cluster.cc:1428] Setting key 4571d9ed0188bd3e637e49e6d20f76cb
I20260925 12:16:23.787793 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } }
I20260925 12:16:23.787966 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 12:16:23.788002 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a52e44ccd14d0995aedbbd60a122b7, State: Initialized, Role: FOLLOWER
I20260925 12:16:23.788160 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } }
I20260925 12:16:23.788285 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 12:16:23.788342 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 12:16:23.788396 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:23.789176 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:23.787222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33471 (local address 127.0.98.62:41829)
0925 12:16:23.787418 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:23.787428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:23.787471 (+ 43us) server_negotiation.cc:407] Connection header received
0925 12:16:23.787752 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:23.787760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:23.787793 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:23.787964 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:23.788623 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:23.788626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:23.788627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:23.788631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:23.788678 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:23.788695 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:23.788769 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:23.788832 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 12:16:23.788913 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260925 12:16:23.789707 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } }
I20260925 12:16:23.789819 22596 leader_election.cc:302] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a52e44ccd14d0995aedbbd60a122b7; no voters:
I20260925 12:16:23.789975 22596 leader_election.cc:288] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 12:16:23.790000 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:23.790098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:23.790163 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [term 1 LEADER]: Becoming Leader. State: Replica: e5a52e44ccd14d0995aedbbd60a122b7, State: Running, Role: LEADER
I20260925 12:16:23.790258 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:23.790242 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } }
I20260925 12:16:23.790542 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } } }
I20260925 12:16:23.790611 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:23.790726 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a52e44ccd14d0995aedbbd60a122b7. Latest consensus state: current_term: 1 leader_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a52e44ccd14d0995aedbbd60a122b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41829 } } }
I20260925 12:16:23.790792 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:23.790939 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:23.791448 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:23.792999 22606 catalog_manager.cc:1347] Generated new cluster ID: 9ac0662ddaa944d88b6ceaffd78003de
I20260925 12:16:23.793054 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:23.798080 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:23.796796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44887 (local address 127.0.98.62:41829)
0925 12:16:23.796833 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:23.796835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:23.797088 (+ 253us) server_negotiation.cc:407] Connection header received
0925 12:16:23.797164 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:23.797165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:23.797171 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:23.797227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:23.797843 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:23.797844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:23.797845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 12:16:23.797849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:23.797864 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 12:16:23.797868 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:23.797924 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:23.797979 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 12:16:23.797995 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 12:16:23.811939 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:23.812114 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:23.818303 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a52e44ccd14d0995aedbbd60a122b7: Generated new TSK 0
I20260925 12:16:23.818490 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:23.825356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22528
2026-09-25T12:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1301 ms)
[----------] 18 tests from AuthTokenIssuingTest (23909 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:23 dist-test-slave-2hqq krb5kdc[22627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:23 dist-test-slave-2hqq krb5kdc[22627](info): set up 2 sockets
Sep 25 12:16:23 dist-test-slave-2hqq krb5kdc[22627](info): commencing operation
krb5kdc: starting...
W20260925 12:16:24.856869 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:24 dist-test-slave-2hqq krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:24.977761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:25.058718 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:25.058951 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:25.058970 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:25.058993 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:25.060210 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:25.060259 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:25.060274 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:25.060287 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:25.060335 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:25.061952 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:25.062240 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:25.062481 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:25.067587 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:25.067646 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:25.067654 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:25.067785 22640 server_base.cc:1034] running on GCE node
I20260925 12:16:25.068041 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:25.068284 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:25.069422 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790338585069405 us; error 38 us; skew 500 ppm
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:25.071838 22640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:25.072350 22640 webserver.cc:460] Webserver started at http://127.0.98.62:42717/ using document root <none> and password file <none>
I20260925 12:16:25.072641 22640 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:25.072710 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:25.072857 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:25.073911 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "1556c910b5774969acae262884fce891"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "71adc249a98cd899e1db0036f9e03ddb"
server_key_iv: "bca623a9460621fb458007f1f0d7ecd1"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.074278 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "1556c910b5774969acae262884fce891"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "71adc249a98cd899e1db0036f9e03ddb"
server_key_iv: "bca623a9460621fb458007f1f0d7ecd1"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.075690 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 12:16:25.076459 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:25.076668 22640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:25.076758 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "1556c910b5774969acae262884fce891"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "71adc249a98cd899e1db0036f9e03ddb"
server_key_iv: "bca623a9460621fb458007f1f0d7ecd1"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.076871 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:25.087045 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:25.088857 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:25.089015 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:25.093103 22640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45629
I20260925 12:16:25.093151 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45629 every 8 connection(s)
I20260925 12:16:25.093518 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:25.094462 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:25.096029 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:25.102876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22640
I20260925 12:16:25.102985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:25.103216 392 external_mini_cluster.cc:1428] Setting key 5b87e86383a6f2b3cbf12a1cd3ca17f1
I20260925 12:16:25.105618 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.096743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.98.62:45629)
0925 12:16:25.096910 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.096914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.096948 (+ 34us) server_negotiation.cc:407] Connection header received
0925 12:16:25.097468 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.097470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.097486 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.097589 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.098090 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.098902 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.099742 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.100037 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.100113 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.102142 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.102148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.102148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.102154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.104083 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.104438 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.104442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.104442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.104575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.104846 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.104847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.104847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.105054 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.105118 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.105335 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.105439 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260925 12:16:25.105644 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.096692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45629 (local address 127.0.0.1:52232)
0925 12:16:25.096831 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.096875 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.097306 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.097615 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.097617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.097854 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.098007 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.098010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.098978 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.098979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.099669 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.099671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.100127 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.100127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.100179 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.100755 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.100763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.102070 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.104167 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.104172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.104174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.104356 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.104653 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.104654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.104654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.104776 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.105122 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.105128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.105206 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.105308 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.105515 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":50,"thread_start_us":67,"threads_started":1}
I20260925 12:16:25.106388 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:25.106772 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.106645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58830)
0925 12:16:25.106694 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.106743 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:16:25.106931 22657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
I20260925 12:16:25.107355 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.107241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58838)
0925 12:16:25.107294 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.107327 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 12:16:25.107527 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:16:25.127982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34441
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:16:25.156812 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } attempt: 1
I20260925 12:16:25.157390 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.157237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58844)
0925 12:16:25.157303 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.157355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 12:16:25.157833 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.157740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58846)
0925 12:16:25.157784 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.157812 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 12:16:25.157987 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
W20260925 12:16:25.209316 22715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:25.209525 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:25.209542 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:25.209563 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:25.210747 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:25.210789 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:25.210803 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:25.210860 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:25.210886 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:25.212390 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:25.212682 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:25.212924 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:25.217767 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:25.217836 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:25.217873 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:25.217862 22715 server_base.cc:1034] running on GCE node
I20260925 12:16:25.218317 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:25.218652 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:25.219820 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790338585219804 us; error 25 us; skew 500 ppm
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:25.222435 22715 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:16:25.222958 22715 webserver.cc:460] Webserver started at http://127.0.98.61:36719/ using document root <none> and password file <none>
I20260925 12:16:25.223215 22715 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:25.223284 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:25.223429 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:25.224552 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data/instance:
uuid: "1ef7458af82c4678b04fe554630d24af"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "e07999023f739f3af1945e1902329167"
server_key_iv: "b7a29e2458221f71250628e37c992f3c"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.224918 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal/instance:
uuid: "1ef7458af82c4678b04fe554630d24af"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "e07999023f739f3af1945e1902329167"
server_key_iv: "b7a29e2458221f71250628e37c992f3c"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.226173 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } attempt: 2
I20260925 12:16:25.226351 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:25.226564 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.226434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58848)
0925 12:16:25.226500 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.226538 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 12:16:25.226917 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.226820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58852)
0925 12:16:25.226868 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.226891 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:16:25.227057 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34441: connect: Connection refused (error 111)
I20260925 12:16:25.227223 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:25.227403 22715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:25.227505 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "1ef7458af82c4678b04fe554630d24af"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "e07999023f739f3af1945e1902329167"
server_key_iv: "b7a29e2458221f71250628e37c992f3c"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.227595 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:25.254715 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:25.256714 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:25.256875 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:25.260721 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34441
I20260925 12:16:25.260759 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34441 every 8 connection(s)
I20260925 12:16:25.261153 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:16:25.262130 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:25.262929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22715
I20260925 12:16:25.263060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-1/wal/instance
I20260925 12:16:25.263273 392 external_mini_cluster.cc:1428] Setting key ca53b3281559b510dbbe74332818bb4d
I20260925 12:16:25.264364 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 12:16:25.274237 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.265259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.0.98.62:45629)
0925 12:16:25.265330 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.265333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.265555 (+ 222us) server_negotiation.cc:407] Connection header received
0925 12:16:25.266009 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.266011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.266027 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.266104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.266716 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.267533 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.268522 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.268857 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.268944 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.271332 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.271333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.271334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.271341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.272799 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.273217 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.273218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.273218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.273276 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.273495 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.273496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.273496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.273602 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.273655 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.273974 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.274080 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 12:16:25.274305 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.265237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45629 (local address 127.0.0.1:52246)
0925 12:16:25.265414 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.265455 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.265908 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.266158 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.266161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.266418 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.266618 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.266622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.267634 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.267634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.268408 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.268410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.268963 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.268963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.269070 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.269645 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.269654 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.271241 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.272874 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.272880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.272882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.273094 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.273333 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.273333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.273334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.273433 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.273659 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.273663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.273833 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.273919 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.274094 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260925 12:16:25.275213 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:25.282764 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.275494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58856)
0925 12:16:25.275547 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.275557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.275600 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.275910 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.275911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.276021 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.276067 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.276068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.277155 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.277156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.277672 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.277674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.278177 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.278178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.278243 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.278751 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.278755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.280000 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.281826 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.281828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.281829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.281989 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.282238 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.282239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.282239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.282288 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.282520 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.282521 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.282557 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.282620 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.282697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 12:16:25.282888 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.275555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.0.98.61:34441)
0925 12:16:25.275737 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.275743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.275766 (+ 23us) server_negotiation.cc:407] Connection header received
0925 12:16:25.275791 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.275794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.275809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.275901 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.276142 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.277068 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.277773 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.278090 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.278169 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.280097 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.280103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.280104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.280109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.281743 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.282086 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.282090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.282090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.282158 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.282344 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.282345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.282345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.282476 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.282536 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.282644 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.282744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260925 12:16:25.283417 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:25.283776 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.283664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60090)
0925 12:16:25.283699 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.283749 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 12:16:25.283927 22732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
I20260925 12:16:25.284346 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.284258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60092)
0925 12:16:25.284305 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.284330 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 12:16:25.284473 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45495: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:16:25.290836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45495
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:16:25.318243 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } attempt: 3
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:25.331429 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.318738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.0.98.61:34441)
0925 12:16:25.319099 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.319101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.319111 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:16:25.319166 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.319168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.319183 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.319870 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.319939 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.320939 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.322270 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.322668 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.324007 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.328455 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.328457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.328458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.328467 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.330182 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.330576 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.330577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.330578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.330638 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.330846 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.330847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.330847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.330989 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.331071 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.331231 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.331313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260925 12:16:25.331429 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.318610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58858)
0925 12:16:25.318708 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.318723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.318824 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.319300 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.319301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.319609 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.319755 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.319759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.321067 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.321068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.322130 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.322133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.322780 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.322781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.322858 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.323427 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.323442 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.328212 (+ 4770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.330294 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.330295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.330295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.330484 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.330710 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.330711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.330711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.330780 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.331072 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.331073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.331122 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.331207 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.331313 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 12:16:25.331888 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:25.332237 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.332136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60104)
0925 12:16:25.332184 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.332215 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 12:16:25.332597 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.332475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60118)
0925 12:16:25.332549 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.332574 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 12:16:25.332701 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45495: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
I20260925 12:16:25.333724 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } attempt: 1
I20260925 12:16:25.334103 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.333983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60122)
0925 12:16:25.334042 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.334076 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 12:16:25.334431 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.334340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60124)
0925 12:16:25.334387 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.334411 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 12:16:25.334532 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45495: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
I20260925 12:16:25.363878 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } attempt: 1
I20260925 12:16:25.364375 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.364232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60140)
0925 12:16:25.364306 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.364345 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 12:16:25.364842 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.364733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60152)
0925 12:16:25.364791 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.364822 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 12:16:25.364964 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45495: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
W20260925 12:16:25.376462 22790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:25.376741 22790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:25.376760 22790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:25.376780 22790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:25.377869 22790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:25.377931 22790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:25.377956 22790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:25.377979 22790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:25.378011 22790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:25.379540 22790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45629,127.0.98.61:34441,127.0.98.60:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:25.379735 22790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:25.379911 22790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:25.384748 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:25.384784 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:25.384887 22790 server_base.cc:1034] running on GCE node
W20260925 12:16:25.384943 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:25.385200 22790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:25.385484 22790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:25.386647 22790 hybrid_clock.cc:648] HybridClock initialized: now 1790338585386642 us; error 36 us; skew 500 ppm
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:25.389091 22790 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:16:25.389564 22790 webserver.cc:460] Webserver started at http://127.0.98.60:42815/ using document root <none> and password file <none>
I20260925 12:16:25.389814 22790 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:25.389883 22790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:25.390029 22790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:25.391070 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data/instance:
uuid: "856e6c4c75c448fcb160c3bf8aa55867"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "10b2914b2a34b8e4a38291725e7b5686"
server_key_iv: "fb3aeb26f290a2b8dc6d002840d9ca99"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.391438 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal/instance:
uuid: "856e6c4c75c448fcb160c3bf8aa55867"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "10b2914b2a34b8e4a38291725e7b5686"
server_key_iv: "fb3aeb26f290a2b8dc6d002840d9ca99"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.392845 22790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:16:25.393575 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:25.393751 22790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:16:25.393853 22790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "856e6c4c75c448fcb160c3bf8aa55867"
format_stamp: "Formatted at 2026-09-25 12:16:25 on dist-test-slave-2hqq"
server_key: "10b2914b2a34b8e4a38291725e7b5686"
server_key_iv: "fb3aeb26f290a2b8dc6d002840d9ca99"
server_key_version: "encryptionkey@0"
I20260925 12:16:25.393949 22790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:25.402678 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } attempt: 2
I20260925 12:16:25.403136 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.402972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60168)
0925 12:16:25.403035 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.403077 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 12:16:25.403509 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.403420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60180)
0925 12:16:25.403461 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.403483 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 12:16:25.403594 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45495: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45495: connect: Connection refused (error 111)
I20260925 12:16:25.404397 22790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:25.406347 22790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:25.406461 22790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:25.410391 22790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45495
I20260925 12:16:25.410437 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45495 every 8 connection(s)
I20260925 12:16:25.410792 22790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:16:25.411777 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:25.413466 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:25.418656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 22790
I20260925 12:16:25.418771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790338290997044-392-0/minicluster-data/master-2/wal/instance
I20260925 12:16:25.418957 392 external_mini_cluster.cc:1428] Setting key 3a98bb61001e92ce89a8bb5874517cac
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:25.422593 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.414168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.0.98.62:45629)
0925 12:16:25.414240 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.414243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.414459 (+ 216us) server_negotiation.cc:407] Connection header received
0925 12:16:25.414895 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.414896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.414906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.414958 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.415531 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.416001 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.416982 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.417162 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.417217 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.419715 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.419716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.419716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.419723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.421231 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.421668 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.421670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.421670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.421733 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.421962 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.421962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.421963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.422109 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.422178 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.422435 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.422527 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 12:16:25.422766 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.414140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45629 (local address 127.0.0.1:52260)
0925 12:16:25.414343 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.414378 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.414815 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.415024 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.415028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.415275 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.415463 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.415467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.416073 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.416074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.416912 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.416915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.417256 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.417257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.417356 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.417872 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.417881 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.419607 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.421311 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.421317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.421319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.421601 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.421801 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.421802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.421802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.421899 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.422174 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.422180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.422291 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.422371 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.422538 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260925 12:16:25.423329 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:25.429116 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.419363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.98.60:45495)
0925 12:16:25.419530 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.419536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.419619 (+ 83us) server_negotiation.cc:407] Connection header received
0925 12:16:25.419698 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.419700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.419714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.419780 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.420608 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.421419 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.422442 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.422994 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.423064 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.425998 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.426005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.426005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.426011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.427742 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.428130 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.428133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.428134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.428195 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.428445 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.428445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.428446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.428652 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.428821 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.428914 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.428993 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260925 12:16:25.431226 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.423601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34441 (local address 127.0.0.1:58870)
0925 12:16:25.423712 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.423721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.423761 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.423891 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.423891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.424001 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.424043 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.424044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.424981 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.424982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.425674 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.425682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.426177 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.426178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.426266 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.426710 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.426715 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.428251 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.430194 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.430194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.430195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.430346 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.430540 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.430540 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.430540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.430578 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.430898 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.430899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.430948 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.431033 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.431127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 12:16:25.431231 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.423658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.0.98.61:34441)
0925 12:16:25.423707 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.423709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.423751 (+ 42us) server_negotiation.cc:407] Connection header received
0925 12:16:25.423805 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.423806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.423817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.423878 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.424116 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.424902 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.425764 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.426108 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.426159 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.428333 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.428335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.428336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.428343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.430123 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.430416 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.430417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.430418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.430478 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.430645 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.430646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.430646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.430793 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.430897 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.431077 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.431147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:16:25.431808 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:25.437542 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.429959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.0.98.62:45629)
0925 12:16:25.429999 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.430001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.430011 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:16:25.430033 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.430034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.430043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.430100 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.430654 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.431257 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.432389 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.432672 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.432750 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.434931 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.434932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.434933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.434941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.436416 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.436775 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.436776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.436776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.436831 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.437022 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.437023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.437024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.437131 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.437330 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.437365 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.437439 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:16:25.439302 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.432036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60200)
0925 12:16:25.432083 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.432092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.432123 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.432234 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.432235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.432393 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.432425 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.432427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.433125 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.433126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.433607 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.433609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.434572 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.434573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.434615 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.435120 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.435125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.436450 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.438315 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.438315 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.438316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.438473 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.438680 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.438680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.438681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.438733 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.438976 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.438976 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.439013 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.439079 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.439176 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 12:16:25.439343 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.432093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.0.98.60:45495)
0925 12:16:25.432125 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.432126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.432134 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:16:25.432169 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.432169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.432189 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.432231 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.432539 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.433070 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.433688 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.434015 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.434082 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.436623 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.436624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.436624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.436631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.438243 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.438555 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.438556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.438556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.438604 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.438802 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.438803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.438803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.438911 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.438983 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.439110 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.439221 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":15}
I20260925 12:16:25.440125 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } attempt: 2
I20260925 12:16:25.441360 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: Bootstrap starting.
I20260925 12:16:25.442425 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:25.442740 22858 log.cc:824] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: Log is configured to *not* fsync() on all Append() calls
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:25.443658 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: No bootstrap required, opened a new log
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:25.445876 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.446040 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856e6c4c75c448fcb160c3bf8aa55867, State: Initialized, Role: FOLLOWER
I20260925 12:16:25.446194 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.446518 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.446602 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:25.446660 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.446806 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.438195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.0.98.61:34441)
0925 12:16:25.438241 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.438243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.438251 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:16:25.438274 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.438274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.438283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.438337 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.438786 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.439260 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.440577 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.440911 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.441000 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.443610 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.443612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.443612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.443620 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.445465 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.445805 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.445806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.445807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.445862 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.446088 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.446089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.446089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.446235 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.446386 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.446615 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.446693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:16:25.448940 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.440397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60206)
0925 12:16:25.440445 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.440454 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.440490 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.440660 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.440661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.440823 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.440867 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.440869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.441870 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.441871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.443188 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.443191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.443670 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.443671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.443732 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.444281 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.444285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.445911 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.447728 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.447729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.447730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.447978 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.448200 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.448201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.448202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.448267 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.448600 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.448601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.448650 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.448732 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.448833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 12:16:25.448975 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.440447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.0.98.60:45495)
0925 12:16:25.440533 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.440535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.440543 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:16:25.440564 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.440565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.440574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.440633 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.440986 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.441775 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.443280 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.443545 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.443607 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.445986 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.445987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.445987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.445994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.447623 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.448068 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.448069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.448069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.448129 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.448333 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.448334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.448334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.448482 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.448563 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.448768 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.448839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 12:16:25.449033 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 12:16:25.449990 22875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:25.450098 22875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:25.450711 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: Bootstrap starting.
I20260925 12:16:25.450057 22682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1556c910b5774969acae262884fce891"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52270
I20260925 12:16:25.451502 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:25.448884 22790 master_runner.cc:186] Error getting master registration for 127.0.98.62:45629: OK, instance_id { permanent_uuid: "1556c910b5774969acae262884fce891" instance_seqno: 1790338585091932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:16:25.451805 22708 log.cc:824] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:25.451822 22790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:16:25.452656 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: No bootstrap required, opened a new log
I20260925 12:16:25.453663 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.453794 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1556c910b5774969acae262884fce891, State: Initialized, Role: FOLLOWER
I20260925 12:16:25.453900 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.454210 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:25.454180 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.454252 22682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 12:16:25.454397 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:25.455221 22682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.455462 22682 leader_election.cc:288] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [CANDIDATE]: Term 1 election: Requested vote from peers 1ef7458af82c4678b04fe554630d24af (127.0.98.61:34441), 856e6c4c75c448fcb160c3bf8aa55867 (127.0.98.60:45495)
I20260925 12:16:25.455571 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.455964 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1556c910b5774969acae262884fce891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "856e6c4c75c448fcb160c3bf8aa55867"
I20260925 12:16:25.456085 22832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:25.457269 22832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1556c910b5774969acae262884fce891 in term 1.
I20260925 12:16:25.457482 22657 leader_election.cc:302] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1556c910b5774969acae262884fce891, 856e6c4c75c448fcb160c3bf8aa55867; no voters:
I20260925 12:16:25.458132 22640 master_runner.cc:186] Error getting master registration for 127.0.98.61:34441: OK, instance_id { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" instance_seqno: 1790338585259634 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:16:25.460438 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:16:25.460614 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:25.460811 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [term 1 LEADER]: Becoming Leader. State: Replica: 1556c910b5774969acae262884fce891, State: Running, Role: LEADER
I20260925 12:16:25.460937 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.461661 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1556c910b5774969acae262884fce891. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.461812 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:25.461229 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:25.462016 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:25.462174 22898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:25.462386 22898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:25.464527 22832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [term 1 FOLLOWER]: Refusing update from remote peer 1556c910b5774969acae262884fce891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:16:25.464933 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:16:25.466240 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1556c910b5774969acae262884fce891. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.466344 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.466696 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.457190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.62:45629)
0925 12:16:25.457232 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.457234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.457368 (+ 134us) server_negotiation.cc:407] Connection header received
0925 12:16:25.457463 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.457464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.457474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.457540 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.458474 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.459202 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.460215 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.460543 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.460612 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.463034 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.463036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.463036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.463044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.465062 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.465675 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.465676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.465677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.465743 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.466007 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.466008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.466009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.466140 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.466250 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.466511 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.466605 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:16:25.467926 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.468014 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:25.468389 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.468467 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.468807 22899 mvcc.cc:205] Tried to move back new op lower bound from 7333226846058786816 to 7333226846049173504. Current Snapshot: MvccSnapshot[applied={T|T < 7333226846058786816}]
I20260925 12:16:25.469316 22900 mvcc.cc:205] Tried to move back new op lower bound from 7333226846058786816 to 7333226846049173504. Current Snapshot: MvccSnapshot[applied={T|T < 7333226846058786816}]
I20260925 12:16:25.469863 22898 catalog_manager.cc:1347] Generated new cluster ID: d78005d443d24f3289a4f980c9bd2fbc
I20260925 12:16:25.469905 22898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:25.478282 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.467062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.0.98.61:34441)
0925 12:16:25.467109 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.467111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.467901 (+ 790us) server_negotiation.cc:407] Connection header received
0925 12:16:25.468043 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.468044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.468066 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.468098 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.469205 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.469964 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.471537 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.472319 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.472368 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.474781 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.474782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.474796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.474804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.476621 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.477214 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.477216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.477216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.477296 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.477575 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.477576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.477576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.477709 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.477789 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.478090 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.478174 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:16:25.480350 22898 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:25.481173 22898 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:25.486757 22898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891: Generated new TSK 0
I20260925 12:16:25.487005 22898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:25.494391 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.467593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.0.98.60:45495)
0925 12:16:25.467631 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.467633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.474409 (+ 6776us) server_negotiation.cc:407] Connection header received
0925 12:16:25.474760 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.474761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.474774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.474808 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.477184 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.477898 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.489138 (+ 11240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.489483 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.489555 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.491320 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.491322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.491322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.491332 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.493168 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.493562 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.493563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.493564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.493642 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.493817 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.493831 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.493831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.493967 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.494065 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.494217 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.494294 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 12:16:25.494711 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } attempt: 3
Sep 25 12:16:25 dist-test-slave-2hqq krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:25.507018 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.494972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45495 (local address 127.0.0.1:60228)
0925 12:16:25.495020 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:25.495031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:25.495065 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:25.495273 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:25.495274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:25.495435 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:25.495483 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.495485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.497928 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.497929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.498758 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:25.498761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.501785 (+ 3024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.501786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:25.501860 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.502293 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:25.502298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:25.503816 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:25.505776 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.505777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.505778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.506014 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.506307 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:25.506307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:25.506308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:25.506388 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:25.506684 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:25.506685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:25.506736 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 12:16:25.506805 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:16:25.506912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:16:25.507025 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:25.495019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.0.98.60:45495)
0925 12:16:25.495140 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.495142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.495149 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:16:25.495178 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.495179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.495188 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.495254 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.495563 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.496351 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.498844 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:25.499156 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:25.499222 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:25.503903 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:25.503904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:25.503904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:25.503911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:25.505704 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.506151 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.506152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.506153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.506238 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:25.506448 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:25.506449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:25.506449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:25.506583 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:25.506658 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:25.506850 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 12:16:25.506929 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 12:16:25.508733 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: Bootstrap starting.
I20260925 12:16:25.509490 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:25.509829 22783 log.cc:824] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:25.510910 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: No bootstrap required, opened a new log
I20260925 12:16:25.513425 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.513582 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef7458af82c4678b04fe554630d24af, State: Initialized, Role: FOLLOWER
I20260925 12:16:25.513671 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } }
I20260925 12:16:25.513979 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:25.514055 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1556c910b5774969acae262884fce891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ef7458af82c4678b04fe554630d24af"
I20260925 12:16:25.514174 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:25.515172 22757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1556c910b5774969acae262884fce891 in term 1.
I20260925 12:16:25.515381 22756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [term 1 FOLLOWER]: Refusing update from remote peer 1556c910b5774969acae262884fce891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:16:25.515946 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 12:16:25.516098 22922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:25.520753 22922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:25.520931 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.521067 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.521153 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1556c910b5774969acae262884fce891. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.521246 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.521310 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:25.521911 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:25.522497 22925 mvcc.cc:205] Tried to move back new op lower bound from 7333226846147305472 to 7333226846049173504. Current Snapshot: MvccSnapshot[applied={T|T < 7333226846058786816}]
W20260925 12:16:25.531805 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.530198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.98.61:34441)
0925 12:16:25.530247 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.530249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.531124 (+ 875us) server_negotiation.cc:407] Connection header received
0925 12:16:25.531228 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.531229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.531239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.531287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.531744 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58890: BlockingRecv error: recv got EOF from 127.0.0.1:58890 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 12:16:25.532577 22862 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.530568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.60:45495)
0925 12:16:25.530608 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.530609 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.532026 (+ 1417us) server_negotiation.cc:407] Connection header received
0925 12:16:25.532124 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.532125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.532135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.532163 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.532453 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60240: BlockingRecv error: recv got EOF from 127.0.0.1:60240 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 12:16:25.533286 22711 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.530864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.0.98.62:45629)
0925 12:16:25.530902 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.530904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.532798 (+ 1894us) server_negotiation.cc:407] Connection header received
0925 12:16:25.532892 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.532894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.532904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.532933 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.533220 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52278: BlockingRecv error: recv got EOF from 127.0.0.1:52278 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 12:16:25.537524 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.536759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.0.98.61:34441)
0925 12:16:25.536794 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.536796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.537003 (+ 207us) server_negotiation.cc:407] Connection header received
0925 12:16:25.537119 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.537120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.537138 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.537166 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.537475 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58906: BlockingRecv error: recv got EOF from 127.0.0.1:58906 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 12:16:25.540616 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.539803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.0.98.61:34441)
0925 12:16:25.539838 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.539839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.540069 (+ 230us) server_negotiation.cc:407] Connection header received
0925 12:16:25.540164 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.540165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.540175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.540204 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.540569 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58916: BlockingRecv error: recv got EOF from 127.0.0.1:58916 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 12:16:25.544598 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.543835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.61:34441)
0925 12:16:25.543867 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.543868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.544077 (+ 209us) server_negotiation.cc:407] Connection header received
0925 12:16:25.544167 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.544168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.544177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.544205 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.544546 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58920: BlockingRecv error: recv got EOF from 127.0.0.1:58920 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 12:16:25.550434 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.549690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.0.98.61:34441)
0925 12:16:25.549724 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.549725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.549944 (+ 219us) server_negotiation.cc:407] Connection header received
0925 12:16:25.550048 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.550049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.550059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.550087 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.550389 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58926: BlockingRecv error: recv got EOF from 127.0.0.1:58926 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 12:16:25.560767 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.559822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.0.98.61:34441)
0925 12:16:25.559868 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.559870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.560134 (+ 264us) server_negotiation.cc:407] Connection header received
0925 12:16:25.560239 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.560240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.560251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.560281 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.560706 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58934: BlockingRecv error: recv got EOF from 127.0.0.1:58934 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 12:16:25.579053 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.578107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.0.98.61:34441)
0925 12:16:25.578151 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.578154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.578437 (+ 283us) server_negotiation.cc:407] Connection header received
0925 12:16:25.578547 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.578548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.578560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.578600 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.578993 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58946: BlockingRecv error: recv got EOF from 127.0.0.1:58946 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 12:16:25.613550 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.612537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.98.61:34441)
0925 12:16:25.612601 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.612603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.612912 (+ 309us) server_negotiation.cc:407] Connection header received
0925 12:16:25.613021 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.613022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.613033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.613064 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.613489 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58960: BlockingRecv error: recv got EOF from 127.0.0.1:58960 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 12:16:25.679771 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.679093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.61:34441)
0925 12:16:25.679168 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.679170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.679181 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:16:25.679260 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.679261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.679278 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.679339 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.679707 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58976: BlockingRecv error: recv got EOF from 127.0.0.1:58976 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 12:16:25.809870 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:25.809102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.0.98.61:34441)
0925 12:16:25.809196 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:25.809198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:25.809269 (+ 71us) server_negotiation.cc:407] Connection header received
0925 12:16:25.809334 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:25.809336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:25.809347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:25.809394 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:25.809772 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58982: BlockingRecv error: recv got EOF from 127.0.0.1:58982 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 12:16:26.003392 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1556c910b5774969acae262884fce891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556c910b5774969acae262884fce891" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45629 } } peers { permanent_uuid: "1ef7458af82c4678b04fe554630d24af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34441 } } peers { permanent_uuid: "856e6c4c75c448fcb160c3bf8aa55867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45495 } } }
I20260925 12:16:26.003574 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1556c910b5774969acae262884fce891 [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:26.067768 22788 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:26.067164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.0.98.61:34441)
0925 12:16:26.067244 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.067246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.067257 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:16:26.067289 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.067289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.067300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.067366 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:26.067686 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58998: BlockingRecv error: recv got EOF from 127.0.0.1:58998 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 12:16:26.450573 22875 catalog_manager.cc:1259] Loaded cluster ID: d78005d443d24f3289a4f980c9bd2fbc
I20260925 12:16:26.450649 22875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: loading cluster ID for follower catalog manager: success
I20260925 12:16:26.451395 22875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: acquiring CA information for follower catalog manager: success
I20260925 12:16:26.451645 22875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 856e6c4c75c448fcb160c3bf8aa55867: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:16:26.521291 22922 catalog_manager.cc:1259] Loaded cluster ID: d78005d443d24f3289a4f980c9bd2fbc
I20260925 12:16:26.521409 22922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: loading cluster ID for follower catalog manager: success
I20260925 12:16:26.522207 22922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: acquiring CA information for follower catalog manager: success
I20260925 12:16:26.522537 22922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ef7458af82c4678b04fe554630d24af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:16:26.583551 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:26.581149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.0.98.61:34441)
0925 12:16:26.581294 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.581296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.581307 (+ 11us) server_negotiation.cc:407] Connection header received
0925 12:16:26.581377 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.581378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.581428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.581514 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:26.581650 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.582136 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.582875 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.583066 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.583119 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:26.583274 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:26.583380 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:26.583438 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:26.583470 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:16:26.583487 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260925 12:16:26.587973 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:26.585475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.0.98.60:45495)
0925 12:16:26.585618 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.585620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.585628 (+ 8us) server_negotiation.cc:407] Connection header received
0925 12:16:26.585666 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.585667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.585701 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.585768 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:26.585842 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.586334 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.587030 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.587459 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.587517 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:26.587697 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:26.587797 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:26.587854 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:26.587892 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 12:16:26.587909 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260925 12:16:26.592654 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:26.589686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.98.61:34441)
0925 12:16:26.589731 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.589733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.589876 (+ 143us) server_negotiation.cc:407] Connection header received
0925 12:16:26.589980 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.589981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.589996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.590046 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:26.590212 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.590847 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.591897 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.592173 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.592234 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:26.592386 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:26.592418 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:26.592471 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:26.592556 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 12:16:26.592581 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:16:26.592742 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:26.589983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.0.98.60:45495)
0925 12:16:26.590025 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.590027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.590276 (+ 249us) server_negotiation.cc:407] Connection header received
0925 12:16:26.590336 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.590337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.590346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.590395 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:26.590514 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.591183 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.591963 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.592197 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.592278 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:26.592428 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:26.592461 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:26.592540 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:26.592632 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 12:16:26.592656 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:16:26.593091 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:26.589689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.98.62:45629)
0925 12:16:26.589806 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:26.589808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:26.589896 (+ 88us) server_negotiation.cc:407] Connection header received
0925 12:16:26.589976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:26.589977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:26.590030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:26.590135 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:26.590243 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.591029 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.592010 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:26.592327 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:26.592405 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:26.592663 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:26.592775 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:26.592823 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:26.592953 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 12:16:26.592979 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260925 12:16:26.594723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22640
I20260925 12:16:26.600275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22715
I20260925 12:16:26.604984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 22790
2026-09-25T12:16:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2781 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 12:16:26 dist-test-slave-2hqq krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 12:16:26 dist-test-slave-2hqq krb5kdc[23012](info): set up 2 sockets
Sep 25 12:16:26 dist-test-slave-2hqq krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260925 12:16:27.636849 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/test-user.keytab.
Sep 25 12:16:27 dist-test-slave-2hqq krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T12:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T12:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 12:16:27.760083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 12:16:27.840736 23025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:27.840947 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:27.840965 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:27.840984 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:27.842051 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:27.842087 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:27.842101 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:27.842114 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:27.842126 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:27.843529 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:27.843716 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:27.843878 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:27.848917 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:27.848927 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:27.848981 23025 server_base.cc:1034] running on GCE node
W20260925 12:16:27.848987 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:27.849331 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:27.849619 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:27.850828 23025 hybrid_clock.cc:648] HybridClock initialized: now 1790338587850814 us; error 43 us; skew 500 ppm
Sep 25 12:16:27 dist-test-slave-2hqq krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:27.860330 23025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 12:16:27.860885 23025 webserver.cc:460] Webserver started at http://127.0.98.62:41191/ using document root <none> and password file <none>
I20260925 12:16:27.861078 23025 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:27.861119 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:27.861287 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:27.862789 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data/instance:
uuid: "332476be480a4716a38ac8763dabfd96"
format_stamp: "Formatted at 2026-09-25 12:16:27 on dist-test-slave-2hqq"
server_key: "fedd89eb136f69ccdfc45088d178216a"
server_key_iv: "653943bc35356f70f0ea516c59d7e1a2"
server_key_version: "encryptionkey@0"
I20260925 12:16:27.863171 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance:
uuid: "332476be480a4716a38ac8763dabfd96"
format_stamp: "Formatted at 2026-09-25 12:16:27 on dist-test-slave-2hqq"
server_key: "fedd89eb136f69ccdfc45088d178216a"
server_key_iv: "653943bc35356f70f0ea516c59d7e1a2"
server_key_version: "encryptionkey@0"
I20260925 12:16:27.864575 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:16:27.865408 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:27.865619 23025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 12:16:27.865715 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal
uuid: "332476be480a4716a38ac8763dabfd96"
format_stamp: "Formatted at 2026-09-25 12:16:27 on dist-test-slave-2hqq"
server_key: "fedd89eb136f69ccdfc45088d178216a"
server_key_iv: "653943bc35356f70f0ea516c59d7e1a2"
server_key_version: "encryptionkey@0"
I20260925 12:16:27.865816 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:27.877632 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:27.879628 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:27.879745 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:27.884478 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39839
I20260925 12:16:27.884575 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39839 every 8 connection(s)
I20260925 12:16:27.884971 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/data/info.pb
I20260925 12:16:27.885954 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:27.887538 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:27 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:27.895063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 23025
I20260925 12:16:27.895176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-0/wal/instance
I20260925 12:16:27.895387 392 external_mini_cluster.cc:1428] Setting key d4f7a3c1394543e6f5ee7aa2fb520b40
I20260925 12:16:27.897549 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.888368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.98.62:39839)
0925 12:16:27.888611 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:27.888616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:27.888654 (+ 38us) server_negotiation.cc:407] Connection header received
0925 12:16:27.889074 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:27.889076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:27.889093 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:27.889174 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:27.889733 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:27.890652 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:27.891521 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:27.891822 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:27.891899 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:27.893975 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:27.893982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:27.893982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:27.893988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:27.895885 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:27.896230 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:27.896233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:27.896233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:27.896365 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:27.896622 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:27.896623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:27.896623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:27.896924 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:27.897017 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:27.897253 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 12:16:27.897387 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":212,"thread_start_us":188,"threads_started":1}
I20260925 12:16:27.897656 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.888212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39839 (local address 127.0.0.1:43304)
0925 12:16:27.888393 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:27.888430 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:27.888934 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:27.889235 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:27.889238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:27.889487 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:27.889640 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:27.889644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:27.890757 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:27.890758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:27.891443 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:27.891445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:27.891917 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:27.891917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:27.891973 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:27.892440 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:27.892449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:27.893890 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:27.895961 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:27.895966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:27.895968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:27.896150 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:27.896432 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:27.896433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:27.896433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:27.896559 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:27.897021 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:27.897028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:27.897109 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 12:16:27.897206 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 12:16:27.897481 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":91,"thread_start_us":64,"threads_started":1}
I20260925 12:16:27.898588 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:27.899027 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.898886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37122)
0925 12:16:27.898945 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:27.898997 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 12:16:27.899209 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
I20260925 12:16:27.899652 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.899533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37126)
0925 12:16:27.899590 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:27.899622 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 12:16:27.899786 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 12:16:27.921634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37049
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:16:27.949036 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } attempt: 1
I20260925 12:16:27.949641 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.949462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37138)
0925 12:16:27.949555 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:27.949609 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 12:16:27.950129 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:27.950024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37146)
0925 12:16:27.950069 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:27.950096 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 12:16:27.950261 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
W20260925 12:16:28.010218 23100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:28.010449 23100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:28.010468 23100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:28.010488 23100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:28.011601 23100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:28.011639 23100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:28.011653 23100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:28.011667 23100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:28.011678 23100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:28.013693 23100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:28.013940 23100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:28.014139 23100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 12:16:28.018474 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } attempt: 2
I20260925 12:16:28.018860 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.018741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37150)
0925 12:16:28.018797 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.018831 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 12:16:28.019148 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:28.019295 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.019192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37164)
0925 12:16:28.019233 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.019258 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 12:16:28.019147 23108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:28.019460 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37049: connect: Connection refused (error 111)
W20260925 12:16:28.019356 23105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:28.019344 23100 server_base.cc:1034] running on GCE node
I20260925 12:16:28.019750 23100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:28.020040 23100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:28.021204 23100 hybrid_clock.cc:648] HybridClock initialized: now 1790338588021178 us; error 52 us; skew 500 ppm
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:28.023528 23100 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 12:16:28.023980 23100 webserver.cc:460] Webserver started at http://127.0.98.61:39233/ using document root <none> and password file <none>
I20260925 12:16:28.024197 23100 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:28.024263 23100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:28.024417 23100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:28.025493 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data/instance:
uuid: "494bc406975440ad87383ff7e37aa82f"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "679fe5901158ef3a17bbf1b4cdd7fc8c"
server_key_iv: "c1aaa6dc67f8e1e76565a8ba3e60c234"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.025846 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal/instance:
uuid: "494bc406975440ad87383ff7e37aa82f"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "679fe5901158ef3a17bbf1b4cdd7fc8c"
server_key_iv: "c1aaa6dc67f8e1e76565a8ba3e60c234"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.027220 23100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 12:16:28.027985 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:28.028156 23100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 12:16:28.028257 23100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal
uuid: "494bc406975440ad87383ff7e37aa82f"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "679fe5901158ef3a17bbf1b4cdd7fc8c"
server_key_iv: "c1aaa6dc67f8e1e76565a8ba3e60c234"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.028329 23100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:28.043418 23100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:28.045617 23100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:28.045778 23100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:28.050031 23100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37049
I20260925 12:16:28.050071 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37049 every 8 connection(s)
I20260925 12:16:28.050451 23100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/data/info.pb
I20260925 12:16:28.051457 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:28.053107 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:28.056788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 23100
I20260925 12:16:28.056910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-1/wal/instance
I20260925 12:16:28.057132 392 external_mini_cluster.cc:1428] Setting key 4db5cfba3b72c5103d91db9ee7fdd6a6
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:28.063753 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.053818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.98.62:39839)
0925 12:16:28.053885 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.053887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.054060 (+ 173us) server_negotiation.cc:407] Connection header received
0925 12:16:28.054506 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.054507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.054518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.054581 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.055222 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.055717 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.056969 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.057263 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.057327 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.060168 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.060169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.060170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.060179 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.062035 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.062593 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.062594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.062595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.062651 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.062988 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.062989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.062990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.063117 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.063196 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.063556 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.063669 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:16:28.064033 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.053733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39839 (local address 127.0.0.1:43320)
0925 12:16:28.053926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.053974 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.054423 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.054658 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.054662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.054914 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.055132 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.055136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.055807 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.055808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.056810 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.056813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.057374 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.057375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.057536 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.058097 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.058107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.060064 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.062121 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.062131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.062134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.062509 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.062727 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.062728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.062728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.062896 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.063199 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.063206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.063397 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.063533 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.063717 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260925 12:16:28.064695 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:28.072818 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.065003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37174)
0925 12:16:28.065099 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.065110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.065147 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.065450 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.065451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.065565 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.065615 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.065617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.066865 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.066865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.067383 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.067384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.067921 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.067922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.067987 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.068458 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.068463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.069844 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.071703 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.071704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.071704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.071870 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.072095 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.072095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.072096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.072150 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.072453 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.072454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.072507 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.072579 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.072683 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 12:16:28.072851 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.065121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.98.61:37049)
0925 12:16:28.065274 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.065280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.065315 (+ 35us) server_negotiation.cc:407] Connection header received
0925 12:16:28.065338 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.065340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.065355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.065432 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.065699 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.066713 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.067458 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.067766 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.067824 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.069947 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.069957 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.069957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.069964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.071598 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.071965 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.071968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.071968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.072027 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.072221 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.072222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.072222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.072374 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.072464 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.072609 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.072734 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260925 12:16:28.073778 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:28.074151 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.074038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45630)
0925 12:16:28.074087 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.074132 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 12:16:28.074280 23117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
I20260925 12:16:28.074676 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.074586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45642)
0925 12:16:28.074630 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.074654 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 12:16:28.074780 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45775: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 12:16:28.083849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODICQW/build/release/bin/kudu
/tmp/dist-test-taskODICQW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45775
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 12:16:28.110647 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } attempt: 3
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:28.119673 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.111187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37186)
0925 12:16:28.111251 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.111263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.111307 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.111477 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.111478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.111661 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.111735 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.111737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.112606 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.112608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.113453 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.113456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.114063 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.114065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.114147 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.114805 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.114828 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.116505 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.118502 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.118503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.118504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.118730 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.118949 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.118949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.118950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.119022 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.119340 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.119341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.119390 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.119461 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.119571 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 12:16:28.119719 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.111254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.0.98.61:37049)
0925 12:16:28.111334 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.111336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.111346 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:16:28.111373 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.111374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.111384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.111451 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.111818 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.112470 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.113560 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.113964 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.114043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.116605 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.116607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.116607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.116616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.118431 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.118803 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.118804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.118805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.118872 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.119117 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.119118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.119119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.119269 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.119353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.119500 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.119590 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 12:16:28.120957 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:28.124001 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } attempt: 1
I20260925 12:16:28.123937 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.121281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45646)
0925 12:16:28.122487 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.122523 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1186}
I20260925 12:16:28.124606 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.124268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45648)
0925 12:16:28.124540 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.124572 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260925 12:16:28.124742 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.124580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45664)
0925 12:16:28.124690 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.124713 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260925 12:16:28.124823 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45775: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
I20260925 12:16:28.124930 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.124836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45666)
0925 12:16:28.124872 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.124899 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 12:16:28.125056 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45775: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
I20260925 12:16:28.155982 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } attempt: 1
I20260925 12:16:28.156948 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.156378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45676)
0925 12:16:28.156458 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.156579 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 12:16:28.157383 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.157276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45688)
0925 12:16:28.157328 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.157357 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 12:16:28.157529 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45775: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
W20260925 12:16:28.172674 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 12:16:28.172910 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 12:16:28.172969 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 12:16:28.173010 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 12:16:28.174129 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 12:16:28.174191 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 12:16:28.174216 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 12:16:28.174255 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 12:16:28.174280 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 12:16:28.175729 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39839,127.0.98.61:37049,127.0.98.60:45775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 11:20:57 UTC on ccycloud.quasar-ptjzfi.root.comops.site
build id 4672270
I20260925 12:16:28.175990 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 12:16:28.176208 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 12:16:28.181001 23181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:28.181001 23184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 12:16:28.181066 23182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 12:16:28.181329 23175 server_base.cc:1034] running on GCE node
I20260925 12:16:28.181634 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 12:16:28.181919 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 12:16:28.183106 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790338588183093 us; error 29 us; skew 500 ppm
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 12:16:28.185510 23175 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 12:16:28.185986 23175 webserver.cc:460] Webserver started at http://127.0.98.60:46079/ using document root <none> and password file <none>
I20260925 12:16:28.186221 23175 fs_manager.cc:359] Metadata directory not provided
I20260925 12:16:28.186288 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 12:16:28.186441 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 12:16:28.187481 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data/instance:
uuid: "1ff57fc4390b46c4a0f880b930cf1677"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "a72d9f8b95e4fd7db91878dba9f43eeb"
server_key_iv: "648c743c38a483054d95c0c0b310391e"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.187844 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal/instance:
uuid: "1ff57fc4390b46c4a0f880b930cf1677"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "a72d9f8b95e4fd7db91878dba9f43eeb"
server_key_iv: "648c743c38a483054d95c0c0b310391e"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.189280 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 12:16:28.190006 23191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 12:16:28.190169 23175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 12:16:28.190269 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal
uuid: "1ff57fc4390b46c4a0f880b930cf1677"
format_stamp: "Formatted at 2026-09-25 12:16:28 on dist-test-slave-2hqq"
server_key: "a72d9f8b95e4fd7db91878dba9f43eeb"
server_key_iv: "648c743c38a483054d95c0c0b310391e"
server_key_version: "encryptionkey@0"
I20260925 12:16:28.190351 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 12:16:28.193187 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } attempt: 2
I20260925 12:16:28.193560 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.193428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45694)
0925 12:16:28.193505 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.193533 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 12:16:28.193902 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.193789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45706)
0925 12:16:28.193844 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.193872 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 12:16:28.194082 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45775: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45775: connect: Connection refused (error 111)
I20260925 12:16:28.221659 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 12:16:28.223582 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 12:16:28.223743 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 12:16:28.227927 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45775
I20260925 12:16:28.227962 23243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45775 every 8 connection(s)
I20260925 12:16:28.228331 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/data/info.pb
I20260925 12:16:28.228868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODICQW/build/release/bin/kudu as pid 23175
I20260925 12:16:28.228964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODICQW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790338290997044-392-0/minicluster-data/master-2/wal/instance
I20260925 12:16:28.229203 392 external_mini_cluster.cc:1428] Setting key 8d07b5a1bfced757933252f183de14c1
I20260925 12:16:28.229511 23244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 12:16:28.232172 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:28.232659 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } attempt: 2
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:28.241303 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.229800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.98.60:45775)
0925 12:16:28.229991 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.230001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.230034 (+ 33us) server_negotiation.cc:407] Connection header received
0925 12:16:28.230692 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.230702 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.230766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.230916 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.231837 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.233215 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.234374 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.234963 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235068 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.237664 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.237671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.237672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.237678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.239774 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240145 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.240148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.240148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.240230 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240434 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.240435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.240436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.240709 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.240893 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.240960 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.241082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260925 12:16:28.241698 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.233031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.0.98.62:39839)
0925 12:16:28.233081 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.233084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.233093 (+ 9us) server_negotiation.cc:407] Connection header received
0925 12:16:28.233194 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.233196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.233206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.233236 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.233779 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.234255 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235259 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.235550 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235608 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.238329 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.238330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.238330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.238337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.240126 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240665 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.240666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.240667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.240721 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240925 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.240926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.240926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.241134 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.241175 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.241419 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.241491 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":26}
I20260925 12:16:28.241717 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.232833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39839 (local address 127.0.0.1:43322)
0925 12:16:28.232995 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.233041 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.233102 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.233325 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.233328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.233632 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.233691 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.233693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.234340 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.234341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.235180 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.235181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.235689 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.235743 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.236191 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.236201 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.238228 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.240236 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.240246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.240571 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.240794 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.240795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.240795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.240854 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.241223 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.241229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.241284 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.241528 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.241639 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260925 12:16:28.242316 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.233132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.0.98.60:45775)
0925 12:16:28.233278 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.233280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.233287 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:16:28.233306 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.233306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.233315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.233380 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.233882 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.234542 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235386 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.235685 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235983 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.238691 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.238692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.238693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.238699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.240931 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.241349 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.241350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.241350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.241418 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.241814 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.241815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.241816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.241924 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.241972 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.242138 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.242198 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260925 12:16:28.242345 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.232990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45726)
0925 12:16:28.233111 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.233120 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.233166 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.233412 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.233413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.233669 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.233754 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.233760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.234629 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.234630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.235298 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.235301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.235784 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.235785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.235850 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.236367 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.236373 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.238613 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.241010 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.241011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.241011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.241275 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.241507 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.241508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.241508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.241582 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.241989 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.241990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.242035 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.242111 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.242216 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":24}
I20260925 12:16:28.242635 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 12:16:28.243852 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: Bootstrap starting.
I20260925 12:16:28.244480 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:28.244822 23093 log.cc:824] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:28.245589 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: No bootstrap required, opened a new log
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 12:16:28.247462 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.247615 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332476be480a4716a38ac8763dabfd96, State: Initialized, Role: FOLLOWER
I20260925 12:16:28.247699 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 12:16:28.248031 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:28.247982 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.248220 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:16:28.250141 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:28.250191 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:28.251762 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.242557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.98.62:39839)
0925 12:16:28.242599 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.242601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.242682 (+ 81us) server_negotiation.cc:407] Connection header received
0925 12:16:28.242775 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.242775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.242783 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.242838 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.243353 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.244005 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.244945 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.245230 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.245266 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.248048 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.248050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.248050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.248056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.249684 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.250828 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.250829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.250830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.250883 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.251162 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.251163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.251163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.251281 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.251384 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.251606 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.251678 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 12:16:28.251798 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.242850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37049 (local address 127.0.0.1:37190)
0925 12:16:28.243349 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.243358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.243414 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.243546 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.243546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.243680 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.243720 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.243721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.244624 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.244625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.245376 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.245379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.245896 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.245897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.245970 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.246354 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.246359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.248653 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.250573 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.250574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.250574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.250780 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.251183 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.251183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.251184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.251249 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.251526 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.251526 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.251566 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.251624 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.251713 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485}
I20260925 12:16:28.252506 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.242955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.0.98.61:37049)
0925 12:16:28.243008 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.243011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.243409 (+ 398us) server_negotiation.cc:407] Connection header received
0925 12:16:28.243476 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.243477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.243488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.243542 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.243794 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.244537 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.245466 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.245775 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.245820 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.248763 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.248764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.248765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.248773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.250497 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.251011 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.251012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.251012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.251105 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.251316 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.251317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.251317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.251441 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.251500 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.252315 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.252399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 12:16:28.251394 23025 master_runner.cc:186] Error getting master registration for 127.0.98.61:37049: OK, instance_id { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" instance_seqno: 1790338588048833 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:16:28.253798 23025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:16:28.254069 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:28.261569 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.252482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.98.61:37049)
0925 12:16:28.252578 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.252580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.252724 (+ 144us) server_negotiation.cc:407] Connection header received
0925 12:16:28.252810 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.252811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.252820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.252847 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.253488 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.254227 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.255107 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.255325 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.255402 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.258280 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.258281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.258282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.258289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.259945 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.260570 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.260572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.260572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.260631 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.260842 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.260843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.260843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.261061 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.261170 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.261379 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.261471 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":72,"threads_started":1}
I20260925 12:16:28.262532 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.254380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.98.60:45775)
0925 12:16:28.254411 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.254413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.254496 (+ 83us) server_negotiation.cc:407] Connection header received
0925 12:16:28.254524 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.254525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.254537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.254602 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.254859 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.255850 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.256933 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.257331 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.257401 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.259215 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.259216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.259217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.259223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.260929 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.261324 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.261325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.261325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.261434 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.261627 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.261628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.261629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.261867 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.261948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.262128 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.262351 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":16}
I20260925 12:16:28.262625 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.254315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45736)
0925 12:16:28.254429 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.254437 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.254484 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.254607 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.254607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.254754 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.254784 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.254785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.255933 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.255933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.256824 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.256826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.257437 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.257438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.257498 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.257877 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.257882 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.259106 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.261012 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.261013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.261013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.261238 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.261488 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.261488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.261489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.261556 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.261971 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.261971 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.262011 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.262080 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.262268 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 12:16:28.263191 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 12:16:28.264156 23067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "332476be480a4716a38ac8763dabfd96"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43330
I20260925 12:16:28.264298 23067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 12:16:28.264261 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: Bootstrap starting.
I20260925 12:16:28.264349 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:28.264961 23244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:28.265249 23244 log.cc:824] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:28.265254 23067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.265491 23067 leader_election.cc:288] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [CANDIDATE]: Term 1 election: Requested vote from peers 494bc406975440ad87383ff7e37aa82f (127.0.98.61:37049), 1ff57fc4390b46c4a0f880b930cf1677 (127.0.98.60:45775)
I20260925 12:16:28.266062 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: No bootstrap required, opened a new log
I20260925 12:16:28.268075 23244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.268232 23244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff57fc4390b46c4a0f880b930cf1677, State: Initialized, Role: FOLLOWER
I20260925 12:16:28.268325 23244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.268697 23244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:28.268700 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.268909 23218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "332476be480a4716a38ac8763dabfd96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ff57fc4390b46c4a0f880b930cf1677"
I20260925 12:16:28.269135 23218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:28.270426 23218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 332476be480a4716a38ac8763dabfd96 in term 1.
I20260925 12:16:28.270682 23042 leader_election.cc:302] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff57fc4390b46c4a0f880b930cf1677, 332476be480a4716a38ac8763dabfd96; no voters:
I20260925 12:16:28.270838 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 12:16:28.271010 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [term 1 LEADER]: Becoming Leader. State: Replica: 332476be480a4716a38ac8763dabfd96, State: Running, Role: LEADER
I20260925 12:16:28.271116 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.269109 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:28.271584 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332476be480a4716a38ac8763dabfd96. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.271664 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: This master's current role is: LEADER
W20260925 12:16:28.272570 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 12:16:28.272630 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 12:16:28.273545 23175 master_runner.cc:186] Error getting master registration for 127.0.98.61:37049: OK, instance_id { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" instance_seqno: 1790338588048833 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 12:16:28.276029 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 12:16:28.280627 23287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 12:16:28.280856 23287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 12:16:28.282871 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.269057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.98.61:37049)
0925 12:16:28.269106 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.269107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.272960 (+ 3853us) server_negotiation.cc:407] Connection header received
0925 12:16:28.273112 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.273113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.273125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.273155 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.274870 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.275594 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.276669 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.276964 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.277007 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.279250 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.279251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.279252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.279259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.281065 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.281511 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.281513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.281513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.281573 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.281787 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.281788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.281789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.281923 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.282445 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.282685 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.282767 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 12:16:28.284397 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.268738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.62:39839)
0925 12:16:28.268796 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.268798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.272857 (+ 4059us) server_negotiation.cc:407] Connection header received
0925 12:16:28.273225 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.273226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.273237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.273268 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.276651 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.277312 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.278251 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.278543 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.278612 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.280940 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.280941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.280942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.280950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.282721 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.283319 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.283320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.283321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.283381 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.283810 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.283811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.283812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.283944 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.284019 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.284209 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.284290 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 12:16:28.285224 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } attempt: 3
I20260925 12:16:28.289489 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.269368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.0.98.60:45775)
0925 12:16:28.269422 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.269424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.281866 (+ 12442us) server_negotiation.cc:407] Connection header received
0925 12:16:28.281971 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.281972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.281983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.282021 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.282694 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.283412 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.284385 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.284702 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.284746 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.286398 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.286400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.286400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.286407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.288127 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.288560 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.288561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.288563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.288620 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.288878 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.288879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.288879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.289007 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.289099 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.289325 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.289403 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 12:16:28 dist-test-slave-2hqq krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790338588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 12:16:28.293303 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.285636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.98.60:45775)
0925 12:16:28.285675 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.285677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.285684 (+ 7us) server_negotiation.cc:407] Connection header received
0925 12:16:28.285703 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.285704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.285712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.285768 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.286085 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.286828 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.287800 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.288128 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.288190 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.290392 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 12:16:28.290393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 12:16:28.290393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 12:16:28.290400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 12:16:28.292102 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.292476 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.292477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.292478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.292591 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 12:16:28.292774 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 12:16:28.292775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 12:16:28.292775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 12:16:28.292889 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 12:16:28.292947 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:28.293152 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 12:16:28.293222 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 12:16:28.293303 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:28.285473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45775 (local address 127.0.0.1:45748)
0925 12:16:28.285519 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 12:16:28.285528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 12:16:28.285564 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 12:16:28.285809 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 12:16:28.285810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 12:16:28.285964 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 12:16:28.286003 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.286005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.286908 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.286909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.287705 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 12:16:28.287707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 12:16:28.288205 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 12:16:28.288207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 12:16:28.288275 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:28.288709 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 12:16:28.288714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 12:16:28.290299 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 12:16:28.292178 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.292179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.292179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.292401 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.292657 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 12:16:28.292658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 12:16:28.292658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 12:16:28.292726 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 12:16:28.292973 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 12:16:28.292974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 12:16:28.293042 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 12:16:28.293106 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 12:16:28.293205 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 12:16:28.296600 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: Bootstrap starting.
I20260925 12:16:28.297367 23218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [term 1 FOLLOWER]: Refusing update from remote peer 332476be480a4716a38ac8763dabfd96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:16:28.297381 23168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: Neither blocks nor log segments found. Creating new log.
I20260925 12:16:28.297704 23168 log.cc:824] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: Log is configured to *not* fsync() on all Append() calls
I20260925 12:16:28.297708 23251 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:16:28.298975 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332476be480a4716a38ac8763dabfd96. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.299063 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:28.299999 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.300088 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:28.300154 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.300235 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:28.301146 23287 catalog_manager.cc:1347] Generated new cluster ID: d42969be062543d68ad2af02d544f20f
I20260925 12:16:28.301203 23287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 12:16:28.304543 23291 mvcc.cc:205] Tried to move back new op lower bound from 7333226857662332928 to 7333226857558323200. Current Snapshot: MvccSnapshot[applied={T|T < 7333226857662332928}]
I20260925 12:16:28.306941 23287 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 12:16:28.307648 23287 catalog_manager.cc:1504] Loading token signing keys...
I20260925 12:16:28.317076 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: No bootstrap required, opened a new log
I20260925 12:16:28.318643 23168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.318800 23168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494bc406975440ad87383ff7e37aa82f, State: Initialized, Role: FOLLOWER
I20260925 12:16:28.318892 23168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } }
I20260925 12:16:28.319197 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.319284 23168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 12:16:28.319396 23141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [term 0 FOLLOWER]: Advancing to term 1
I20260925 12:16:28.320448 23141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [term 1 FOLLOWER]: Refusing update from remote peer 332476be480a4716a38ac8763dabfd96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 12:16:28.320719 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:28.320914 23278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 12:16:28.320551 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "332476be480a4716a38ac8763dabfd96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "494bc406975440ad87383ff7e37aa82f"
I20260925 12:16:28.322047 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 332476be480a4716a38ac8763dabfd96 in term 1.
I20260925 12:16:28.324509 23306 catalog_manager.cc:1259] Loaded cluster ID: d42969be062543d68ad2af02d544f20f
I20260925 12:16:28.324563 23306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: loading cluster ID for follower catalog manager: success
I20260925 12:16:28.325706 23306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: acquiring CA information for follower catalog manager: success
W20260925 12:16:28.325815 23306 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 12:16:28.329370 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.329478 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96 [sys.catalog]: This master's current role is: LEADER
I20260925 12:16:28.329854 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.329981 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 12:16:28.330075 23287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332476be480a4716a38ac8763dabfd96: Generated new TSK 0
I20260925 12:16:28.330283 23287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 12:16:28.330631 23309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332476be480a4716a38ac8763dabfd96. Latest consensus state: current_term: 1 leader_uuid: "332476be480a4716a38ac8763dabfd96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332476be480a4716a38ac8763dabfd96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } peers { permanent_uuid: "494bc406975440ad87383ff7e37aa82f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37049 } } peers { permanent_uuid: "1ff57fc4390b46c4a0f880b930cf1677" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45775 } } }
I20260925 12:16:28.330838 23309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 12:16:28.362149 23096 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.360431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.62:39839)
0925 12:16:28.360484 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.360486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.361471 (+ 985us) server_negotiation.cc:407] Connection header received
0925 12:16:28.361584 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.361585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.361596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.361627 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.362076 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43360: BlockingRecv error: recv got EOF from 127.0.0.1:43360 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 12:16:28.363089 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.360882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.0.98.61:37049)
0925 12:16:28.360921 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.360923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.362529 (+ 1606us) server_negotiation.cc:407] Connection header received
0925 12:16:28.362628 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.362629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.362640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.362669 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.363025 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37218: BlockingRecv error: recv got EOF from 127.0.0.1:37218 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:16:28.363981 23247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.361180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.0.98.60:45775)
0925 12:16:28.361222 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.361224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.363469 (+ 2245us) server_negotiation.cc:407] Connection header received
0925 12:16:28.363593 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.363594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.363604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.363634 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.363919 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45754: BlockingRecv error: recv got EOF from 127.0.0.1:45754 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 12:16:28.366330 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.365598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.98.61:37049)
0925 12:16:28.365635 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.365636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.365841 (+ 205us) server_negotiation.cc:407] Connection header received
0925 12:16:28.365949 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.365950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.365970 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.365999 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.366283 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37234: BlockingRecv error: recv got EOF from 127.0.0.1:37234 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 12:16:28.369761 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.368957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.98.61:37049)
0925 12:16:28.369009 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.369011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.369231 (+ 220us) server_negotiation.cc:407] Connection header received
0925 12:16:28.369324 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.369325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.369335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.369364 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.369710 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37246: BlockingRecv error: recv got EOF from 127.0.0.1:37246 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 12:16:28.373849 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.373076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.61:37049)
0925 12:16:28.373109 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.373111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.373326 (+ 215us) server_negotiation.cc:407] Connection header received
0925 12:16:28.373450 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.373451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.373462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.373491 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.373790 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37254: BlockingRecv error: recv got EOF from 127.0.0.1:37254 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 12:16:28.379901 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.379094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.0.98.61:37049)
0925 12:16:28.379130 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.379132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.379380 (+ 248us) server_negotiation.cc:407] Connection header received
0925 12:16:28.379492 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.379493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.379503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.379531 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.379852 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37262: BlockingRecv error: recv got EOF from 127.0.0.1:37262 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 12:16:28.390305 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.389379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.61:37049)
0925 12:16:28.389424 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.389426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.389706 (+ 280us) server_negotiation.cc:407] Connection header received
0925 12:16:28.389813 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.389814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.389825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.389855 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.390244 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37278: BlockingRecv error: recv got EOF from 127.0.0.1:37278 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 12:16:28.408948 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.407897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.0.98.61:37049)
0925 12:16:28.407944 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.407946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.408249 (+ 303us) server_negotiation.cc:407] Connection header received
0925 12:16:28.408374 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.408375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.408386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.408417 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.408886 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37284: BlockingRecv error: recv got EOF from 127.0.0.1:37284 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 12:16:28.443580 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.442568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.98.61:37049)
0925 12:16:28.442615 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.442618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.442903 (+ 285us) server_negotiation.cc:407] Connection header received
0925 12:16:28.443011 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.443012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.443026 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.443059 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.443520 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37292: BlockingRecv error: recv got EOF from 127.0.0.1:37292 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 12:16:28.510232 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.509350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.0.98.61:37049)
0925 12:16:28.509448 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.509451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.509558 (+ 107us) server_negotiation.cc:407] Connection header received
0925 12:16:28.509660 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.509661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.509674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.509749 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.510158 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37302: BlockingRecv error: recv got EOF from 127.0.0.1:37302 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 12:16:28.640051 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.639390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.0.98.61:37049)
0925 12:16:28.639460 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.639462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.639566 (+ 104us) server_negotiation.cc:407] Connection header received
0925 12:16:28.639629 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.639630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.639641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.639704 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.640005 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37312: BlockingRecv error: recv got EOF from 127.0.0.1:37312 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 12:16:28.898052 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 12:16:28.897429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.61:37049)
0925 12:16:28.897507 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:28.897509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:28.897521 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:16:28.897597 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:28.897598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:28.897608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:28.897671 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 12:16:28.898006 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37324: BlockingRecv error: recv got EOF from 127.0.0.1:37324 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 12:16:29.273077 23286 catalog_manager.cc:1259] Loaded cluster ID: d42969be062543d68ad2af02d544f20f
I20260925 12:16:29.273164 23286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: loading cluster ID for follower catalog manager: success
I20260925 12:16:29.273954 23286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: acquiring CA information for follower catalog manager: success
I20260925 12:16:29.274197 23286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ff57fc4390b46c4a0f880b930cf1677: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:16:29.326392 23306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 494bc406975440ad87383ff7e37aa82f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 12:16:29.414479 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:29.411392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.61:37049)
0925 12:16:29.411585 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:29.411588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:29.411600 (+ 12us) server_negotiation.cc:407] Connection header received
0925 12:16:29.411699 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:29.411699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:29.411766 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:29.411862 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:29.412039 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.412670 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.413536 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.413779 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.413871 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:29.414064 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:29.414193 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:29.414294 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:29.414326 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 12:16:29.414350 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260925 12:16:29.419212 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:29.416512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.0.98.60:45775)
0925 12:16:29.416632 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:29.416634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:29.416663 (+ 29us) server_negotiation.cc:407] Connection header received
0925 12:16:29.416748 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:29.416749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:29.416784 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:29.416853 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:29.416981 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.417491 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.418261 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.418534 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.418598 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:29.418808 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:29.418966 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:29.419051 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:29.419084 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 12:16:29.419135 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260925 12:16:29.424304 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:29.421332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.0.98.61:37049)
0925 12:16:29.421379 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:29.421382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:29.421476 (+ 94us) server_negotiation.cc:407] Connection header received
0925 12:16:29.421581 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:29.421582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:29.421594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:29.421631 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:29.421783 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.422557 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423573 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.423805 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423864 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:29.424013 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:29.424046 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:29.424111 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:29.424199 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 12:16:29.424222 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 12:16:29.424329 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:29.421193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.98.62:39839)
0925 12:16:29.421357 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:29.421360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:29.421370 (+ 10us) server_negotiation.cc:407] Connection header received
0925 12:16:29.421416 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:29.421417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:29.421469 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:29.421550 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:29.421707 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.422342 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423428 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.423728 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423793 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:29.423991 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:29.424106 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:29.424170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:29.424219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 12:16:29.424247 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260925 12:16:29.424355 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 12:16:29.421603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.0.98.60:45775)
0925 12:16:29.421638 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 12:16:29.421640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 12:16:29.421718 (+ 78us) server_negotiation.cc:407] Connection header received
0925 12:16:29.421822 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 12:16:29.421823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 12:16:29.421834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 12:16:29.421883 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 12:16:29.422014 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.422701 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423555 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 12:16:29.423836 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 12:16:29.423870 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 12:16:29.424100 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 12:16:29.424132 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 12:16:29.424161 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 12:16:29.424274 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 12:16:29.424294 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 12:16:29.426064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 23025
I20260925 12:16:29.431562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 23100
I20260925 12:16:29.436713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODICQW/build/release/bin/kudu with pid 23175
2026-09-25T12:16:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2832 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5613 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298446 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS